From rochkind at jhu.edu Thu Jun 4 14:58:09 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 04 Jun 2009 14:58:09 -0400 Subject: [Blacklight-development] question: new installation Message-ID: <4A281941.3090003@jhu.edu> So we're finally going ahead and doing a fresh installation of Blacklight locally, to start looking at, developing, and configuring it. Hooray. I believe there's a Blacklight wiki somewhere maybe, but I'm having trouble finding it on Google? What I'm really looking for is a pointer to the most up-to-date instructions for installing Blacklight ( and the SolrMarc indexer in concert). Can someone help point me to the right place? Also, would you reccommend that we install trunk, or the latest tagged release, to start messing with? I'd lean toward trunk? Jonathan From eos8d at virginia.edu Thu Jun 4 16:19:38 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Thu, 4 Jun 2009 16:19:38 -0400 Subject: [Blacklight-development] question: new installation In-Reply-To: <4A281941.3090003@jhu.edu> References: <4A281941.3090003@jhu.edu> Message-ID: <1041E85B-F22F-4154-924D-30BAB0BBECD2@virginia.edu> Hi, Jonathan. You should be able to find everything you need on http://blacklightopac.org/ If you're planning to keep up with development, the easiest way to go is to check out the demo app from trunk. Instructions are here: http://wiki.blacklightopac.org/doku.php?id=installing_the_demo_app Make sure you also install all the pre-requisites, which are listed here: http://wiki.blacklightopac.org/doku.php?id=pre-requisites_and_architecture The code from trunk now comes distributed with solrmarc, along with rake tasks for invoking it. I haven't actually tried them (anyone else want to report?), but they're slated for the next release, which will hopefully happen early next week. I also just added a capistrano task that you can modify for local deployment use. Good luck and let us know how the installation goes and whether there's anything we can do to make it easier. Bess On Jun 4, 2009, at 2:58 PM, Jonathan Rochkind wrote: > So we're finally going ahead and doing a fresh installation of > Blacklight locally, to start looking at, developing, and configuring > it. > Hooray. > > I believe there's a Blacklight wiki somewhere maybe, but I'm having > trouble finding it on Google? What I'm really looking for is a pointer > to the most up-to-date instructions for installing Blacklight ( and > the > SolrMarc indexer in concert). Can someone help point me to the right > place? > > Also, would you reccommend that we install trunk, or the latest tagged > release, to start messing with? I'd lean toward trunk? > > Jonathan > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From ndushay at stanford.edu Thu Jun 4 16:43:10 2009 From: ndushay at stanford.edu (Naomi Dushay) Date: Thu, 4 Jun 2009 13:43:10 -0700 Subject: [Blacklight-development] question: new installation In-Reply-To: <4A281941.3090003@jhu.edu> References: <4A281941.3090003@jhu.edu> Message-ID: <60B2EA46-13FF-4C3B-AC92-C9D63080AFC9@stanford.edu> Currently: http://blacklightopac.org is all you need to remember. http://wiki.blacklightopac.org is the wiki (but there's a link to it in the left nav of the above). The trunk had a ton of changes in the last 14 days. Stanford is just coming up to date today. To see what's in the trunk, and not in the 2.1 release, check out the roadmap: http://blacklightopac.org:8080/jira/browse/CODEBASE?report=com.atlassian.jira.plugin.system.project%3Aroadmap-panel We'll probably get a 2.2 release out in the next week or so. - Naomi On Jun 4, 2009, at 11:58 AM, Jonathan Rochkind wrote: > So we're finally going ahead and doing a fresh installation of > Blacklight locally, to start looking at, developing, and configuring > it. Hooray. > > I believe there's a Blacklight wiki somewhere maybe, but I'm having > trouble finding it on Google? What I'm really looking for is a > pointer to the most up-to-date instructions for installing > Blacklight ( and the SolrMarc indexer in concert). Can someone help > point me to the right place? > > Also, would you reccommend that we install trunk, or the latest > tagged release, to start messing with? I'd lean toward trunk? > > Jonathan > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Mon Jun 15 12:02:37 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Mon, 15 Jun 2009 12:02:37 -0400 Subject: [Blacklight-development] dependencies Message-ID: <4A36709D.6030804@jhu.edu> Okay, looks like the dependencies documentation is now stored in rdoc instead of the wiki? That's where the link on blacklightopac.org took me. The dependency list at: http://blacklight.rubyforge.org/files/vendor/plugins/blacklight/PRE-REQUISITES_rdoc.html Does not specify a version of Rails. But does provide an _example_ of 'sudo gem install rails ?version ?2.3.2?' Is 2.3.2 in fact the current supported version of Rails? Or is any version of Rails supported? (I kinda doubt this, Rails version tends to be a dependency). Jonathan From eos8d at virginia.edu Mon Jun 15 14:39:30 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Mon, 15 Jun 2009 14:39:30 -0400 Subject: [Blacklight-development] dependencies In-Reply-To: <4A36709D.6030804@jhu.edu> References: <4A36709D.6030804@jhu.edu> Message-ID: Hey, Jonathan. It should be 2.3 or above. I'll update that in the docs. And yes, we have moved to storing the dependencies and installation docs in rdoc now. Bess On Jun 15, 2009, at 12:02 PM, Jonathan Rochkind wrote: > Okay, looks like the dependencies documentation is now stored in rdoc > instead of the wiki? That's where the link on blacklightopac.org > took me. > > The dependency list at: > > http://blacklight.rubyforge.org/files/vendor/plugins/blacklight/PRE-REQUISITES_rdoc.html > > Does not specify a version of Rails. But does provide an _example_ of > 'sudo gem install rails ?version ?2.3.2?' > > Is 2.3.2 in fact the current supported version of Rails? Or is any > version of Rails supported? (I kinda doubt this, Rails version tends > to > be a dependency). > > Jonathan > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Mon Jun 15 15:52:48 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Mon, 15 Jun 2009 15:52:48 -0400 Subject: [Blacklight-development] installing demo, can't db:migrate Message-ID: <4A36A690.9000803@jhu.edu> I get this error when I try to run a 'rake db:migrate' on a freshly checked out copy of the demo app. Any advice? I'm kind of at a loss, I'm not sure why it's looking for hanna/rdoctask in order to run a db:migrate, and the stack trace isn't helping me. [rochkind at xs001 rails]$ rake db:migrate --trace (in /opt/blacklight/bl-demo/rails) rake aborted! no such file to load -- hanna/rdoctask /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require' /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:156:in `require' /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:521:in `new_constants_in' /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:156:in `require' /opt/blacklight/bl-demo/rails/Rakefile:46 /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2383:in `load' /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2383:in `raw_load_rakefile' /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2017:in `load_rakefile' /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling' /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2016:in `load_rakefile' /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2000:in `run' /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling' /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in `run' /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/bin/rake:31 /usr/bin/rake:19:in `load' /usr/bin/rake:19 [rochkind at xs001 rails]$ From rochkind at jhu.edu Mon Jun 15 15:58:51 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Mon, 15 Jun 2009 15:58:51 -0400 Subject: [Blacklight-development] installing demo, can't db:migrate In-Reply-To: <4A36A690.9000803@jhu.edu> References: <4A36A690.9000803@jhu.edu> Message-ID: <4A36A7FB.7030502@jhu.edu> Ah, it's becuase the demo Rakefile includes the line: require 'hanna/rdoctask' Should this be removed? Grepping for this mysterious 'hanna' thing (is that someone's username?) everywhere, it's in several more places in the 'demo' app. Including hard-coded paths that definitely aren't on my system, and seem like they shouldn't be in the demo app. Should this be here? I can fix this, if I am sure I know I'm fixing it right and it needs fixing. But I have no idea what this hanna is. [rochkind at xs001 rails]$ grep -r hanna * Rakefile: require 'hanna/rdoctask' vendor/plugins/blacklight/vendor/gems/authlogic-2.0.13/Rakefile:ENV['RDOCOPT'] = "-S -f html -T hanna" vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/Rakefile: hanna_dir = '/Users/mislav/Projects/Hanna/lib' vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/Rakefile: $:.unshift hanna_dir if File.exists? hanna_dir vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/Rakefile: require 'hanna/rdoctask' Jonathan Rochkind wrote: > I get this error when I try to run a 'rake db:migrate' on a freshly > checked out copy of the demo app. Any advice? I'm kind of at a loss, > I'm not sure why it's looking for hanna/rdoctask in order to run a > db:migrate, and the stack trace isn't helping me. > > [rochkind at xs001 rails]$ rake db:migrate --trace > (in /opt/blacklight/bl-demo/rails) > rake aborted! > no such file to load -- hanna/rdoctask > /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in > `gem_original_require' > /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' > /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:156:in > `require' > /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:521:in > `new_constants_in' > /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:156:in > `require' > /opt/blacklight/bl-demo/rails/Rakefile:46 > /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2383:in `load' > /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2383:in > `raw_load_rakefile' > /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2017:in `load_rakefile' > /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in > `standard_exception_handling' > /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2016:in `load_rakefile' > /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2000:in `run' > /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in > `standard_exception_handling' > /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in `run' > /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/bin/rake:31 > /usr/bin/rake:19:in `load' > /usr/bin/rake:19 > [rochkind at xs001 rails]$ > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From eos8d at virginia.edu Mon Jun 15 17:07:00 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Mon, 15 Jun 2009 17:07:00 -0400 Subject: [Blacklight-development] installing demo, can't db:migrate In-Reply-To: <4A36A7FB.7030502@jhu.edu> References: <4A36A690.9000803@jhu.edu> <4A36A7FB.7030502@jhu.edu> Message-ID: Hi, Jonathan. Dang, thanks for finding that. Just do "gem install hanna" and it should fix it. Hanna is the rdoc generator I'm using for the rdoc rake task. It gives you nicer rdoc options, and is how I'm creating the files at http://blacklight.rubyforge.org I didn't think it would require it in order to run the migrations, though. Guess I need to add it to the list of required gems. Bess On Jun 15, 2009, at 3:58 PM, Jonathan Rochkind wrote: > Ah, it's becuase the demo Rakefile includes the line: > > require 'hanna/rdoctask' > > Should this be removed? > > Grepping for this mysterious 'hanna' thing (is that someone's > username?) > everywhere, it's in several more places in the 'demo' app. Including > hard-coded paths that definitely aren't on my system, and seem like > they > shouldn't be in the demo app. > > Should this be here? I can fix this, if I am sure I know I'm fixing it > right and it needs fixing. But I have no idea what this hanna is. > > > [rochkind at xs001 rails]$ grep -r hanna * > Rakefile: require 'hanna/rdoctask' > vendor/plugins/blacklight/vendor/gems/authlogic-2.0.13/ > Rakefile:ENV['RDOCOPT'] > = "-S -f html -T hanna" > vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/ > Rakefile: > hanna_dir = '/Users/mislav/Projects/Hanna/lib' > vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/ > Rakefile: > $:.unshift hanna_dir if File.exists? hanna_dir > vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/ > Rakefile: > require 'hanna/rdoctask' > > > > Jonathan Rochkind wrote: >> I get this error when I try to run a 'rake db:migrate' on a freshly >> checked out copy of the demo app. Any advice? I'm kind of at a loss, >> I'm not sure why it's looking for hanna/rdoctask in order to run a >> db:migrate, and the stack trace isn't helping me. >> >> [rochkind at xs001 rails]$ rake db:migrate --trace >> (in /opt/blacklight/bl-demo/rails) >> rake aborted! >> no such file to load -- hanna/rdoctask >> /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in >> `gem_original_require' >> /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in >> `require' >> /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/ >> dependencies.rb:156:in >> `require' >> /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/ >> dependencies.rb:521:in >> `new_constants_in' >> /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/ >> dependencies.rb:156:in >> `require' >> /opt/blacklight/bl-demo/rails/Rakefile:46 >> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2383:in `load' >> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2383:in >> `raw_load_rakefile' >> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2017:in >> `load_rakefile' >> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in >> `standard_exception_handling' >> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2016:in >> `load_rakefile' >> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2000:in `run' >> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in >> `standard_exception_handling' >> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in `run' >> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/bin/rake:31 >> /usr/bin/rake:19:in `load' >> /usr/bin/rake:19 >> [rochkind at xs001 rails]$ >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Mon Jun 15 17:13:20 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Mon, 15 Jun 2009 17:13:20 -0400 Subject: [Blacklight-development] installing demo, can't db:migrate In-Reply-To: References: <4A36A690.9000803@jhu.edu> <4A36A7FB.7030502@jhu.edu> Message-ID: <4A36B970.10201@jhu.edu> Yeah, it looks like it requires it in order to run any rake task at all, since when loading the Rakefile class, it runs into a 'require' statement that involves hanna. It looks like I actually have commit rights. Do you want me to add that as a dependency in the pre-req file in the 'demo' trunk and commit it? No problem, just asking before committing. I'm still kind of suspicious of lines in demo like: Rakefile: hanna_dir = '/Users/mislav/Projects/Hanna/lib' Since that's definitely not a directory that I have. But I guess I'll wait and see if it causes a problem. Jonathan Bess Sadler wrote: > Hi, Jonathan. > > Dang, thanks for finding that. Just do "gem install hanna" and it > should fix it. Hanna is the rdoc generator I'm using for the rdoc rake > task. It gives you nicer rdoc options, and is how I'm creating the > files at http://blacklight.rubyforge.org > > I didn't think it would require it in order to run the migrations, > though. Guess I need to add it to the list of required gems. > > Bess > > On Jun 15, 2009, at 3:58 PM, Jonathan Rochkind wrote: > > >> Ah, it's becuase the demo Rakefile includes the line: >> >> require 'hanna/rdoctask' >> >> Should this be removed? >> >> Grepping for this mysterious 'hanna' thing (is that someone's >> username?) >> everywhere, it's in several more places in the 'demo' app. Including >> hard-coded paths that definitely aren't on my system, and seem like >> they >> shouldn't be in the demo app. >> >> Should this be here? I can fix this, if I am sure I know I'm fixing it >> right and it needs fixing. But I have no idea what this hanna is. >> >> >> [rochkind at xs001 rails]$ grep -r hanna * >> Rakefile: require 'hanna/rdoctask' >> vendor/plugins/blacklight/vendor/gems/authlogic-2.0.13/ >> Rakefile:ENV['RDOCOPT'] >> = "-S -f html -T hanna" >> vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/ >> Rakefile: >> hanna_dir = '/Users/mislav/Projects/Hanna/lib' >> vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/ >> Rakefile: >> $:.unshift hanna_dir if File.exists? hanna_dir >> vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/ >> Rakefile: >> require 'hanna/rdoctask' >> >> >> >> Jonathan Rochkind wrote: >> >>> I get this error when I try to run a 'rake db:migrate' on a freshly >>> checked out copy of the demo app. Any advice? I'm kind of at a loss, >>> I'm not sure why it's looking for hanna/rdoctask in order to run a >>> db:migrate, and the stack trace isn't helping me. >>> >>> [rochkind at xs001 rails]$ rake db:migrate --trace >>> (in /opt/blacklight/bl-demo/rails) >>> rake aborted! >>> no such file to load -- hanna/rdoctask >>> /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in >>> `gem_original_require' >>> /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in >>> `require' >>> /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/ >>> dependencies.rb:156:in >>> `require' >>> /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/ >>> dependencies.rb:521:in >>> `new_constants_in' >>> /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/ >>> dependencies.rb:156:in >>> `require' >>> /opt/blacklight/bl-demo/rails/Rakefile:46 >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2383:in `load' >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2383:in >>> `raw_load_rakefile' >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2017:in >>> `load_rakefile' >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in >>> `standard_exception_handling' >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2016:in >>> `load_rakefile' >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2000:in `run' >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in >>> `standard_exception_handling' >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in `run' >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/bin/rake:31 >>> /usr/bin/rake:19:in `load' >>> /usr/bin/rake:19 >>> [rochkind at xs001 rails]$ >>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From rochkind at jhu.edu Mon Jun 15 17:15:14 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Mon, 15 Jun 2009 17:15:14 -0400 Subject: [Blacklight-development] installing demo, can't db:migrate In-Reply-To: References: <4A36A690.9000803@jhu.edu> <4A36A7FB.7030502@jhu.edu> Message-ID: <4A36B9E2.6040900@jhu.edu> Where does hanna come from? It doesn't appear to be in the gem repos I've configured per install instructions. [rochkind at xs001 rails]$ gem sources *** CURRENT SOURCES *** http://gems.rubyforge.org/ http://gems.rubyonrails.org [rochkind at xs001 rails]$ sudo gem install hanna ERROR: could not find gem hanna locally or in a repository Bess Sadler wrote: > Hi, Jonathan. > > Dang, thanks for finding that. Just do "gem install hanna" and it > should fix it. Hanna is the rdoc generator I'm using for the rdoc rake > task. It gives you nicer rdoc options, and is how I'm creating the > files at http://blacklight.rubyforge.org > > I didn't think it would require it in order to run the migrations, > though. Guess I need to add it to the list of required gems. > > Bess > > On Jun 15, 2009, at 3:58 PM, Jonathan Rochkind wrote: > > >> Ah, it's becuase the demo Rakefile includes the line: >> >> require 'hanna/rdoctask' >> >> Should this be removed? >> >> Grepping for this mysterious 'hanna' thing (is that someone's >> username?) >> everywhere, it's in several more places in the 'demo' app. Including >> hard-coded paths that definitely aren't on my system, and seem like >> they >> shouldn't be in the demo app. >> >> Should this be here? I can fix this, if I am sure I know I'm fixing it >> right and it needs fixing. But I have no idea what this hanna is. >> >> >> [rochkind at xs001 rails]$ grep -r hanna * >> Rakefile: require 'hanna/rdoctask' >> vendor/plugins/blacklight/vendor/gems/authlogic-2.0.13/ >> Rakefile:ENV['RDOCOPT'] >> = "-S -f html -T hanna" >> vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/ >> Rakefile: >> hanna_dir = '/Users/mislav/Projects/Hanna/lib' >> vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/ >> Rakefile: >> $:.unshift hanna_dir if File.exists? hanna_dir >> vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/ >> Rakefile: >> require 'hanna/rdoctask' >> >> >> >> Jonathan Rochkind wrote: >> >>> I get this error when I try to run a 'rake db:migrate' on a freshly >>> checked out copy of the demo app. Any advice? I'm kind of at a loss, >>> I'm not sure why it's looking for hanna/rdoctask in order to run a >>> db:migrate, and the stack trace isn't helping me. >>> >>> [rochkind at xs001 rails]$ rake db:migrate --trace >>> (in /opt/blacklight/bl-demo/rails) >>> rake aborted! >>> no such file to load -- hanna/rdoctask >>> /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in >>> `gem_original_require' >>> /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in >>> `require' >>> /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/ >>> dependencies.rb:156:in >>> `require' >>> /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/ >>> dependencies.rb:521:in >>> `new_constants_in' >>> /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/ >>> dependencies.rb:156:in >>> `require' >>> /opt/blacklight/bl-demo/rails/Rakefile:46 >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2383:in `load' >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2383:in >>> `raw_load_rakefile' >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2017:in >>> `load_rakefile' >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in >>> `standard_exception_handling' >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2016:in >>> `load_rakefile' >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2000:in `run' >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in >>> `standard_exception_handling' >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in `run' >>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/bin/rake:31 >>> /usr/bin/rake:19:in `load' >>> /usr/bin/rake:19 >>> [rochkind at xs001 rails]$ >>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From eos8d at virginia.edu Mon Jun 15 17:23:20 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Mon, 15 Jun 2009 17:23:20 -0400 Subject: [Blacklight-development] mislav-hanna In-Reply-To: <4A36B9E2.6040900@jhu.edu> References: <4A36A690.9000803@jhu.edu> <4A36A7FB.7030502@jhu.edu> <4A36B9E2.6040900@jhu.edu> Message-ID: <91A030F4-EB6D-4A22-B92C-86B42FA9489B@virginia.edu> Hey, Jonathan. Did you add github as one of your gem sources? Hanna is here: http://github.com/mislav/hanna/tree/master And I gave you the wrong command earlier, sorry. The right one is: sudo gem install mislav-hanna Maybe we bundle this gem? Bess On Jun 15, 2009, at 5:15 PM, Jonathan Rochkind wrote: > Where does hanna come from? It doesn't appear to be in the gem repos > I've configured per install instructions. > > [rochkind at xs001 rails]$ gem sources > *** CURRENT SOURCES *** > > http://gems.rubyforge.org/ > http://gems.rubyonrails.org > > [rochkind at xs001 rails]$ sudo gem install hanna > ERROR: could not find gem hanna locally or in a repository > > > Bess Sadler wrote: >> Hi, Jonathan. >> >> Dang, thanks for finding that. Just do "gem install hanna" and it >> should fix it. Hanna is the rdoc generator I'm using for the rdoc >> rake >> task. It gives you nicer rdoc options, and is how I'm creating the >> files at http://blacklight.rubyforge.org >> >> I didn't think it would require it in order to run the migrations, >> though. Guess I need to add it to the list of required gems. >> >> Bess >> >> On Jun 15, 2009, at 3:58 PM, Jonathan Rochkind wrote: >> >> >>> Ah, it's becuase the demo Rakefile includes the line: >>> >>> require 'hanna/rdoctask' >>> >>> Should this be removed? >>> >>> Grepping for this mysterious 'hanna' thing (is that someone's >>> username?) >>> everywhere, it's in several more places in the 'demo' app. Including >>> hard-coded paths that definitely aren't on my system, and seem like >>> they >>> shouldn't be in the demo app. >>> >>> Should this be here? I can fix this, if I am sure I know I'm >>> fixing it >>> right and it needs fixing. But I have no idea what this hanna is. >>> >>> >>> [rochkind at xs001 rails]$ grep -r hanna * >>> Rakefile: require 'hanna/rdoctask' >>> vendor/plugins/blacklight/vendor/gems/authlogic-2.0.13/ >>> Rakefile:ENV['RDOCOPT'] >>> = "-S -f html -T hanna" >>> vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/ >>> Rakefile: >>> hanna_dir = '/Users/mislav/Projects/Hanna/lib' >>> vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/ >>> Rakefile: >>> $:.unshift hanna_dir if File.exists? hanna_dir >>> vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/ >>> Rakefile: >>> require 'hanna/rdoctask' >>> >>> >>> >>> Jonathan Rochkind wrote: >>> >>>> I get this error when I try to run a 'rake db:migrate' on a freshly >>>> checked out copy of the demo app. Any advice? I'm kind of at a >>>> loss, >>>> I'm not sure why it's looking for hanna/rdoctask in order to run a >>>> db:migrate, and the stack trace isn't helping me. >>>> >>>> [rochkind at xs001 rails]$ rake db:migrate --trace >>>> (in /opt/blacklight/bl-demo/rails) >>>> rake aborted! >>>> no such file to load -- hanna/rdoctask >>>> /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in >>>> `gem_original_require' >>>> /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in >>>> `require' >>>> /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/ >>>> dependencies.rb:156:in >>>> `require' >>>> /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/ >>>> dependencies.rb:521:in >>>> `new_constants_in' >>>> /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/ >>>> dependencies.rb:156:in >>>> `require' >>>> /opt/blacklight/bl-demo/rails/Rakefile:46 >>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2383:in `load' >>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2383:in >>>> `raw_load_rakefile' >>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2017:in >>>> `load_rakefile' >>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in >>>> `standard_exception_handling' >>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2016:in >>>> `load_rakefile' >>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2000:in `run' >>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in >>>> `standard_exception_handling' >>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in `run' >>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/bin/rake:31 >>>> /usr/bin/rake:19:in `load' >>>> /usr/bin/rake:19 >>>> [rochkind at xs001 rails]$ >>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Mon Jun 15 17:26:31 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Mon, 15 Jun 2009 17:26:31 -0400 Subject: [Blacklight-development] mislav-hanna In-Reply-To: <91A030F4-EB6D-4A22-B92C-86B42FA9489B@virginia.edu> References: <4A36A690.9000803@jhu.edu> <4A36A7FB.7030502@jhu.edu> <4A36B9E2.6040900@jhu.edu> <91A030F4-EB6D-4A22-B92C-86B42FA9489B@virginia.edu> Message-ID: <4A36BC87.9070806@jhu.edu> Yeah, jaron helped me out with that. I got it installed. Need github as a gem source (so that needs to be added to instructions), and need 'gem install mislav-hanna'. I'd add that to the pre-requisites doc... but I can't find it anywhere in the source! It is somewhere in the source now, right? Where do I find the source of that pre-requisites doc? Bess Sadler wrote: > Hey, Jonathan. > > Did you add github as one of your gem sources? Hanna is here: > > http://github.com/mislav/hanna/tree/master > > And I gave you the wrong command earlier, sorry. The right one is: > > sudo gem install mislav-hanna > > Maybe we bundle this gem? > > Bess > > On Jun 15, 2009, at 5:15 PM, Jonathan Rochkind wrote: > > >> Where does hanna come from? It doesn't appear to be in the gem repos >> I've configured per install instructions. >> >> [rochkind at xs001 rails]$ gem sources >> *** CURRENT SOURCES *** >> >> http://gems.rubyforge.org/ >> http://gems.rubyonrails.org >> >> [rochkind at xs001 rails]$ sudo gem install hanna >> ERROR: could not find gem hanna locally or in a repository >> >> >> Bess Sadler wrote: >> >>> Hi, Jonathan. >>> >>> Dang, thanks for finding that. Just do "gem install hanna" and it >>> should fix it. Hanna is the rdoc generator I'm using for the rdoc >>> rake >>> task. It gives you nicer rdoc options, and is how I'm creating the >>> files at http://blacklight.rubyforge.org >>> >>> I didn't think it would require it in order to run the migrations, >>> though. Guess I need to add it to the list of required gems. >>> >>> Bess >>> >>> On Jun 15, 2009, at 3:58 PM, Jonathan Rochkind wrote: >>> >>> >>> >>>> Ah, it's becuase the demo Rakefile includes the line: >>>> >>>> require 'hanna/rdoctask' >>>> >>>> Should this be removed? >>>> >>>> Grepping for this mysterious 'hanna' thing (is that someone's >>>> username?) >>>> everywhere, it's in several more places in the 'demo' app. Including >>>> hard-coded paths that definitely aren't on my system, and seem like >>>> they >>>> shouldn't be in the demo app. >>>> >>>> Should this be here? I can fix this, if I am sure I know I'm >>>> fixing it >>>> right and it needs fixing. But I have no idea what this hanna is. >>>> >>>> >>>> [rochkind at xs001 rails]$ grep -r hanna * >>>> Rakefile: require 'hanna/rdoctask' >>>> vendor/plugins/blacklight/vendor/gems/authlogic-2.0.13/ >>>> Rakefile:ENV['RDOCOPT'] >>>> = "-S -f html -T hanna" >>>> vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/ >>>> Rakefile: >>>> hanna_dir = '/Users/mislav/Projects/Hanna/lib' >>>> vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/ >>>> Rakefile: >>>> $:.unshift hanna_dir if File.exists? hanna_dir >>>> vendor/plugins/blacklight/vendor/gems/mislav-will_paginate-2.3.8/ >>>> Rakefile: >>>> require 'hanna/rdoctask' >>>> >>>> >>>> >>>> Jonathan Rochkind wrote: >>>> >>>> >>>>> I get this error when I try to run a 'rake db:migrate' on a freshly >>>>> checked out copy of the demo app. Any advice? I'm kind of at a >>>>> loss, >>>>> I'm not sure why it's looking for hanna/rdoctask in order to run a >>>>> db:migrate, and the stack trace isn't helping me. >>>>> >>>>> [rochkind at xs001 rails]$ rake db:migrate --trace >>>>> (in /opt/blacklight/bl-demo/rails) >>>>> rake aborted! >>>>> no such file to load -- hanna/rdoctask >>>>> /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in >>>>> `gem_original_require' >>>>> /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in >>>>> `require' >>>>> /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/ >>>>> dependencies.rb:156:in >>>>> `require' >>>>> /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/ >>>>> dependencies.rb:521:in >>>>> `new_constants_in' >>>>> /usr/lib/ruby/gems/1.8/gems/activesupport-2.3.2/lib/active_support/ >>>>> dependencies.rb:156:in >>>>> `require' >>>>> /opt/blacklight/bl-demo/rails/Rakefile:46 >>>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2383:in `load' >>>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2383:in >>>>> `raw_load_rakefile' >>>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2017:in >>>>> `load_rakefile' >>>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in >>>>> `standard_exception_handling' >>>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2016:in >>>>> `load_rakefile' >>>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2000:in `run' >>>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in >>>>> `standard_exception_handling' >>>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in `run' >>>>> /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/bin/rake:31 >>>>> /usr/bin/rake:19:in `load' >>>>> /usr/bin/rake:19 >>>>> [rochkind at xs001 rails]$ >>>>> >>>>> _______________________________________________ >>>>> Blacklight-development mailing list >>>>> Blacklight-development at rubyforge.org >>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From eos8d at virginia.edu Tue Jun 16 09:36:57 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Tue, 16 Jun 2009 09:36:57 -0400 Subject: [Blacklight-development] installing demo, can't db:migrate In-Reply-To: <4A36B970.10201@jhu.edu> References: <4A36A690.9000803@jhu.edu> <4A36A7FB.7030502@jhu.edu> <4A36B970.10201@jhu.edu> Message-ID: <1C6675A9-A310-40C1-B9AF-11D101898C85@virginia.edu> On Jun 15, 2009, at 5:13 PM, Jonathan Rochkind wrote: > It looks like I actually have commit rights. You do? I'm replying off list here. I'm looking at rubyforge and I don't see you as one of the developers. Can you try to commit something in the docs and let me know if you're able to? Bess From eos8d at virginia.edu Tue Jun 16 09:50:52 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Tue, 16 Jun 2009 09:50:52 -0400 Subject: [Blacklight-development] installing demo, can't db:migrate In-Reply-To: <1C6675A9-A310-40C1-B9AF-11D101898C85@virginia.edu> References: <4A36A690.9000803@jhu.edu> <4A36A7FB.7030502@jhu.edu> <4A36B970.10201@jhu.edu> <1C6675A9-A310-40C1-B9AF-11D101898C85@virginia.edu> Message-ID: <0FFA2745-1D96-4EF3-9760-96B42A49920F@virginia.edu> d'oh! I guess I'm not replying off list. One day I'll master basic emailing skills. Sorry, folks. Bess On Jun 16, 2009, at 9:36 AM, Bess Sadler wrote: > > On Jun 15, 2009, at 5:13 PM, Jonathan Rochkind wrote: > >> It looks like I actually have commit rights. > > You do? I'm replying off list here. I'm looking at rubyforge and I > don't see you as one of the developers. Can you try to commit > something in the docs and let me know if you're able to? > > Bess > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Thu Jun 18 11:55:39 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 18 Jun 2009 11:55:39 -0400 Subject: [Blacklight-development] problems starting solr from 'demo' app Message-ID: <4A3A637B.3010404@jhu.edu> Going through the demo instructions, trying to start solr with jetty as included with the demo app. Run into the following. Any ideas or advice would be appreciated. 18-Jun-09 11:53:22 AM org.apache.solr.servlet.SolrDispatchFilter init SEVERE: Could not start SOLR. Check solr/home property java.lang.NoClassDefFoundError: org.apache.solr.search.FastLRUCache [stack trace omitted] and... 2009-06-18 11:53:22.681::WARN: failed SolrRequestFilter java.lang.NoClassDefFoundError: org.apache.solr.core.SolrCore [stack trace omitted] [And some more, but I'm guessing they're caused by the first ones, perhaps all caused by the missing FastLRUCache ? ] From rochkind at jhu.edu Thu Jun 18 12:58:20 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 18 Jun 2009 12:58:20 -0400 Subject: [Blacklight-development] problems starting solr from 'demo' app In-Reply-To: <763570460906180930t2cc632e5k9462ff2d18534e75@mail.gmail.com> References: <4A3A637B.3010404@jhu.edu> <763570460906180930t2cc632e5k9462ff2d18534e75@mail.gmail.com> Message-ID: <4A3A722C.80305@jhu.edu> Oh, do I need to install solr myself seperately? I was thinking it came with the 'demo' app. I haven't done any install of Solr myself, I just checked out the 'demo' app, and then tried to run "cd jetty && java -jar start.jar" as instructed. Is there an install step I've missed for the demo app? Jason Ronallo wrote: > What version of Solr are you running? I think FastLRU Cache only came to be in trunk/1.4. We're running 1.3 multicore so I had to change that to the older LRUCache. Didn't cause any problems that I can see. > > Jason > > On Thu, Jun 18, 2009 at 11:55 AM, Jonathan Rochkind > wrote: > Going through the demo instructions, trying to start solr with jetty as included with the demo app. Run into the following. Any ideas or advice would be appreciated. > > 18-Jun-09 11:53:22 AM org.apache.solr.servlet.SolrDispatchFilter init > SEVERE: Could not start SOLR. Check solr/home property > java.lang.NoClassDefFoundError: org.apache.solr.search.FastLRUCache > > [stack trace omitted] > > and... > > 2009-06-18 11:53:22.681::WARN: failed SolrRequestFilter > java.lang.NoClassDefFoundError: org.apache.solr.core.SolrCore > > [stack trace omitted] > > > [And some more, but I'm guessing they're caused by the first ones, perhaps all caused by the missing FastLRUCache ? ] > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > From jamie at dang.com Thu Jun 18 12:49:50 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Thu, 18 Jun 2009 12:49:50 -0400 Subject: [Blacklight-development] problems starting solr from 'demo' app In-Reply-To: <4A3A637B.3010404@jhu.edu> References: <4A3A637B.3010404@jhu.edu> Message-ID: <4B3B9878-DF22-4438-8C63-6D565358BC96@dang.com> OS: Java version: Specific instructions you're using: Other significant data points: On Jun 18, 2009, at 11:55 AM, Jonathan Rochkind wrote: > Going through the demo instructions, trying to start solr with jetty > as included with the demo app. Run into the following. Any ideas or > advice would be appreciated. > > 18-Jun-09 11:53:22 AM org.apache.solr.servlet.SolrDispatchFilter init > SEVERE: Could not start SOLR. Check solr/home property > java.lang.NoClassDefFoundError: org.apache.solr.search.FastLRUCache > > [stack trace omitted] > > and... > > 2009-06-18 11:53:22.681::WARN: failed SolrRequestFilter > java.lang.NoClassDefFoundError: org.apache.solr.core.SolrCore > > [stack trace omitted] > > > [And some more, but I'm guessing they're caused by the first ones, > perhaps all caused by the missing FastLRUCache ? ] > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Thu Jun 18 13:41:19 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 18 Jun 2009 13:41:19 -0400 Subject: [Blacklight-development] problems starting solr from 'demo' app In-Reply-To: <4B3B9878-DF22-4438-8C63-6D565358BC96@dang.com> References: <4A3A637B.3010404@jhu.edu> <4B3B9878-DF22-4438-8C63-6D565358BC96@dang.com> Message-ID: <4A3A7C3F.9020605@jhu.edu> RHEL 5 java version "1.4.2" gij (GNU libgcj) version 4.1.2 20080704 (Red Hat 4.1.2-44) Do you think we need the Sun java instead of the Red Hat supplied GNU java? That's what my sysadmin Tony suggests maybe? Using the instructions at: http://blacklight.rubyforge.org/files/DEMO_README.html. Jamie Orchard-Hays wrote: > OS: > Java version: > Specific instructions you're using: > Other significant data points: > > On Jun 18, 2009, at 11:55 AM, Jonathan Rochkind wrote: > > >> Going through the demo instructions, trying to start solr with jetty >> as included with the demo app. Run into the following. Any ideas or >> advice would be appreciated. >> >> 18-Jun-09 11:53:22 AM org.apache.solr.servlet.SolrDispatchFilter init >> SEVERE: Could not start SOLR. Check solr/home property >> java.lang.NoClassDefFoundError: org.apache.solr.search.FastLRUCache >> >> [stack trace omitted] >> >> and... >> >> 2009-06-18 11:53:22.681::WARN: failed SolrRequestFilter >> java.lang.NoClassDefFoundError: org.apache.solr.core.SolrCore >> >> [stack trace omitted] >> >> >> [And some more, but I'm guessing they're caused by the first ones, >> perhaps all caused by the missing FastLRUCache ? ] >> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From jamie at dang.com Thu Jun 18 13:43:17 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Thu, 18 Jun 2009 13:43:17 -0400 Subject: [Blacklight-development] problems starting solr from 'demo' app In-Reply-To: <4A3A722C.80305@jhu.edu> References: <4A3A637B.3010404@jhu.edu> <763570460906180930t2cc632e5k9462ff2d18534e75@mail.gmail.com> <4A3A722C.80305@jhu.edu> Message-ID: <2BB0D13F-5B1B-4415-AAC5-276255285A61@dang.com> It's in there. That's the correct instruction to start it. On Jun 18, 2009, at 12:58 PM, Jonathan Rochkind wrote: > Oh, do I need to install solr myself seperately? I was thinking it > came with the 'demo' app. I haven't done any install of Solr myself, > I just checked out the 'demo' app, and then tried to run "cd jetty > && java -jar start.jar" as instructed. > > Is there an install step I've missed for the demo app? > > Jason Ronallo wrote: >> What version of Solr are you running? I think FastLRU Cache only >> came to be in trunk/1.4. We're running 1.3 multicore so I had to >> change that to the older LRUCache. Didn't cause any problems that I >> can see. >> >> Jason >> >> On Thu, Jun 18, 2009 at 11:55 AM, Jonathan Rochkind >> > wrote: >> Going through the demo instructions, trying to start solr with >> jetty as included with the demo app. Run into the following. Any >> ideas or advice would be appreciated. >> >> 18-Jun-09 11:53:22 AM org.apache.solr.servlet.SolrDispatchFilter init >> SEVERE: Could not start SOLR. Check solr/home property >> java.lang.NoClassDefFoundError: org.apache.solr.search.FastLRUCache >> >> [stack trace omitted] >> >> and... >> >> 2009-06-18 11:53:22.681::WARN: failed SolrRequestFilter >> java.lang.NoClassDefFoundError: org.apache.solr.core.SolrCore >> >> [stack trace omitted] >> >> >> [And some more, but I'm guessing they're caused by the first ones, >> perhaps all caused by the missing FastLRUCache ? ] >> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org> > >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> >> >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From jamie at dang.com Thu Jun 18 14:13:09 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Thu, 18 Jun 2009 14:13:09 -0400 Subject: [Blacklight-development] problems starting solr from 'demo' app In-Reply-To: <4A3A7C3F.9020605@jhu.edu> References: <4A3A637B.3010404@jhu.edu> <4B3B9878-DF22-4438-8C63-6D565358BC96@dang.com> <4A3A7C3F.9020605@jhu.edu> Message-ID: <83C74BEE-A798-48FC-99D7-D745415A981B@dang.com> "Solr requires Java 1.5 ": http://wiki.apache.org/solr/FAQ#head-d0235ca79aa77fd73507d37b82462abd2a848a46 Also, I'd get Sun's version on. IBM's might work, but we know Sun's does. On Jun 18, 2009, at 1:41 PM, Jonathan Rochkind wrote: > RHEL 5 > java version "1.4.2" > gij (GNU libgcj) version 4.1.2 20080704 (Red Hat 4.1.2-44) > > Do you think we need the Sun java instead of the Red Hat supplied > GNU java? That's what my sysadmin Tony suggests maybe? > > Using the instructions at: http://blacklight.rubyforge.org/files/DEMO_README.html > . > > > Jamie Orchard-Hays wrote: >> OS: >> Java version: >> Specific instructions you're using: >> Other significant data points: >> >> On Jun 18, 2009, at 11:55 AM, Jonathan Rochkind wrote: >> >> >>> Going through the demo instructions, trying to start solr with >>> jetty as included with the demo app. Run into the following. Any >>> ideas or advice would be appreciated. >>> >>> 18-Jun-09 11:53:22 AM org.apache.solr.servlet.SolrDispatchFilter >>> init >>> SEVERE: Could not start SOLR. Check solr/home property >>> java.lang.NoClassDefFoundError: org.apache.solr.search.FastLRUCache >>> >>> [stack trace omitted] >>> >>> and... >>> >>> 2009-06-18 11:53:22.681::WARN: failed SolrRequestFilter >>> java.lang.NoClassDefFoundError: org.apache.solr.core.SolrCore >>> >>> [stack trace omitted] >>> >>> >>> [And some more, but I'm guessing they're caused by the first >>> ones, perhaps all caused by the missing FastLRUCache ? ] >>> >>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Thu Jun 18 14:14:24 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 18 Jun 2009 14:14:24 -0400 Subject: [Blacklight-development] problems starting solr from 'demo' app In-Reply-To: <83C74BEE-A798-48FC-99D7-D745415A981B@dang.com> References: <4A3A637B.3010404@jhu.edu> <4B3B9878-DF22-4438-8C63-6D565358BC96@dang.com> <4A3A7C3F.9020605@jhu.edu> <83C74BEE-A798-48FC-99D7-D745415A981B@dang.com> Message-ID: <4A3A8400.3060907@jhu.edu> Thanks Jamie! I'll submit a match to the PRE-REQUISITES doc to link to the solr requirements, and make the reccommendation to use Sun Java too. Jonathan Jamie Orchard-Hays wrote: > "Solr requires Java 1.5 ": http://wiki.apache.org/solr/FAQ#head-d0235ca79aa77fd73507d37b82462abd2a848a46 > > Also, I'd get Sun's version on. IBM's might work, but we know Sun's > does. > > > On Jun 18, 2009, at 1:41 PM, Jonathan Rochkind wrote: > > >> RHEL 5 >> java version "1.4.2" >> gij (GNU libgcj) version 4.1.2 20080704 (Red Hat 4.1.2-44) >> >> Do you think we need the Sun java instead of the Red Hat supplied >> GNU java? That's what my sysadmin Tony suggests maybe? >> >> Using the instructions at: http://blacklight.rubyforge.org/files/DEMO_README.html >> . >> >> >> Jamie Orchard-Hays wrote: >> >>> OS: >>> Java version: >>> Specific instructions you're using: >>> Other significant data points: >>> >>> On Jun 18, 2009, at 11:55 AM, Jonathan Rochkind wrote: >>> >>> >>> >>>> Going through the demo instructions, trying to start solr with >>>> jetty as included with the demo app. Run into the following. Any >>>> ideas or advice would be appreciated. >>>> >>>> 18-Jun-09 11:53:22 AM org.apache.solr.servlet.SolrDispatchFilter >>>> init >>>> SEVERE: Could not start SOLR. Check solr/home property >>>> java.lang.NoClassDefFoundError: org.apache.solr.search.FastLRUCache >>>> >>>> [stack trace omitted] >>>> >>>> and... >>>> >>>> 2009-06-18 11:53:22.681::WARN: failed SolrRequestFilter >>>> java.lang.NoClassDefFoundError: org.apache.solr.core.SolrCore >>>> >>>> [stack trace omitted] >>>> >>>> >>>> [And some more, but I'm guessing they're caused by the first >>>> ones, perhaps all caused by the missing FastLRUCache ? ] >>>> >>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From erikhatcher at mac.com Thu Jun 18 13:58:42 2009 From: erikhatcher at mac.com (Erik Hatcher) Date: Thu, 18 Jun 2009 13:58:42 -0400 Subject: [Blacklight-development] problems starting solr from 'demo' app In-Reply-To: <4A3A7C3F.9020605@jhu.edu> References: <4A3A637B.3010404@jhu.edu> <4B3B9878-DF22-4438-8C63-6D565358BC96@dang.com> <4A3A7C3F.9020605@jhu.edu> Message-ID: <018B1AFD-630B-4593-B74A-9195604A3B77@mac.com> Solr requires Java 1.5+ Erik On Jun 18, 2009, at 1:41 PM, Jonathan Rochkind wrote: > RHEL 5 > java version "1.4.2" > gij (GNU libgcj) version 4.1.2 20080704 (Red Hat 4.1.2-44) > > Do you think we need the Sun java instead of the Red Hat supplied > GNU java? That's what my sysadmin Tony suggests maybe? > > Using the instructions at: http://blacklight.rubyforge.org/files/DEMO_README.html > . > > > Jamie Orchard-Hays wrote: >> OS: >> Java version: >> Specific instructions you're using: >> Other significant data points: >> >> On Jun 18, 2009, at 11:55 AM, Jonathan Rochkind wrote: >> >> >>> Going through the demo instructions, trying to start solr with >>> jetty as included with the demo app. Run into the following. Any >>> ideas or advice would be appreciated. >>> >>> 18-Jun-09 11:53:22 AM org.apache.solr.servlet.SolrDispatchFilter >>> init >>> SEVERE: Could not start SOLR. Check solr/home property >>> java.lang.NoClassDefFoundError: org.apache.solr.search.FastLRUCache >>> >>> [stack trace omitted] >>> >>> and... >>> >>> 2009-06-18 11:53:22.681::WARN: failed SolrRequestFilter >>> java.lang.NoClassDefFoundError: org.apache.solr.core.SolrCore >>> >>> [stack trace omitted] >>> >>> >>> [And some more, but I'm guessing they're caused by the first >>> ones, perhaps all caused by the missing FastLRUCache ? ] >>> >>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Thu Jun 18 16:30:53 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 18 Jun 2009 16:30:53 -0400 Subject: [Blacklight-development] problems starting solr from 'demo' app In-Reply-To: <018B1AFD-630B-4593-B74A-9195604A3B77@mac.com> References: <4A3A637B.3010404@jhu.edu> <4B3B9878-DF22-4438-8C63-6D565358BC96@dang.com> <4A3A7C3F.9020605@jhu.edu> <018B1AFD-630B-4593-B74A-9195604A3B77@mac.com> Message-ID: <4A3AA3FD.3000408@jhu.edu> And for those playing the home game, that indeed took care of my issue. Thanks! Onward. Erik Hatcher wrote: > Solr requires Java 1.5+ > > Erik > > On Jun 18, 2009, at 1:41 PM, Jonathan Rochkind wrote: > > >> RHEL 5 >> java version "1.4.2" >> gij (GNU libgcj) version 4.1.2 20080704 (Red Hat 4.1.2-44) >> >> Do you think we need the Sun java instead of the Red Hat supplied >> GNU java? That's what my sysadmin Tony suggests maybe? >> >> Using the instructions at: http://blacklight.rubyforge.org/files/DEMO_README.html >> . >> >> >> Jamie Orchard-Hays wrote: >> >>> OS: >>> Java version: >>> Specific instructions you're using: >>> Other significant data points: >>> >>> On Jun 18, 2009, at 11:55 AM, Jonathan Rochkind wrote: >>> >>> >>> >>>> Going through the demo instructions, trying to start solr with >>>> jetty as included with the demo app. Run into the following. Any >>>> ideas or advice would be appreciated. >>>> >>>> 18-Jun-09 11:53:22 AM org.apache.solr.servlet.SolrDispatchFilter >>>> init >>>> SEVERE: Could not start SOLR. Check solr/home property >>>> java.lang.NoClassDefFoundError: org.apache.solr.search.FastLRUCache >>>> >>>> [stack trace omitted] >>>> >>>> and... >>>> >>>> 2009-06-18 11:53:22.681::WARN: failed SolrRequestFilter >>>> java.lang.NoClassDefFoundError: org.apache.solr.core.SolrCore >>>> >>>> [stack trace omitted] >>>> >>>> >>>> [And some more, but I'm guessing they're caused by the first >>>> ones, perhaps all caused by the missing FastLRUCache ? ] >>>> >>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From aplacilla at jhu.edu Thu Jun 18 17:07:15 2009 From: aplacilla at jhu.edu (Tony Placilla) Date: Thu, 18 Jun 2009 17:07:15 -0400 Subject: [Blacklight-development] problems starting solr from 'demo' app In-Reply-To: <4A3AA3FD.3000408@jhu.edu> References: <4A3A637B.3010404@jhu.edu> <4B3B9878-DF22-4438-8C63-6D565358BC96@dang.com> <4A3A7C3F.9020605@jhu.edu> <018B1AFD-630B-4593-B74A-9195604A3B77@mac.com> <4A3AA3FD.3000408@jhu.edu> Message-ID: <2AC333D68D50E241AFA6FF41ACD5E65A12B255F5D7@JHEMTEXVS1.win.ad.jhu.edu> >From the FWIW dept. As Jonathan mentioned I am *not* a fan of using code compiled & installed from source. And being as we're a RedHat shop I need it to play nice with RPM for package management. Java was a piece of cake. yum install java-1.5.0-sun.x86_64 & Bjorn Stronginthearm's your uncle! The Ruby issue was a bit more complex (but not too) & it remains to be seen if the version we've got installed will provide all the tools needed. 1.8.7 doesn't exist in an RH repository other than Rawhide. Thank you, but no. Fedora 10, however, has ruby-1.8.6 in an SRPM wget ftp://ftp.pbone.net/mirror/download.fedora.redhat.com/pub/fedora/linux/releases/10/Everything/source/SRPMS/ruby-1.8.6.287-2.fc10.src.rpm yum install tcl-devel tk-devel libX11-devel rpmbuild --rebuild ruby-1.8.6.287-2.fc10.src.rpm cd /usr/src/redhat/RPMS/x86_64/ & install the binaries (assuming of course that you have removed the RHEL supplied ones) Then wget ftp://ftp.pbone.net/mirror/download.fedora.redhat.com/pub/fedora/linux/releases/10/Everything/source/SRPMS/rubygems-1.2.0-2.fc10.src.rpm lather, rinse, repeat as above Tony Placilla Sr. UNIX Systems Administrator The Sheridan Libraries Johns Hopkins University -----Original Message----- From: blacklight-development-bounces at rubyforge.org [mailto:blacklight-development-bounces at rubyforge.org] On Behalf Of Jonathan Rochkind Sent: Thursday, June 18, 2009 4:31 PM To: blacklight-development at rubyforge.org Subject: Re: [Blacklight-development] problems starting solr from 'demo' app And for those playing the home game, that indeed took care of my issue. Thanks! Onward. Erik Hatcher wrote: > Solr requires Java 1.5+ > > Erik > > On Jun 18, 2009, at 1:41 PM, Jonathan Rochkind wrote: > > >> RHEL 5 >> java version "1.4.2" >> gij (GNU libgcj) version 4.1.2 20080704 (Red Hat 4.1.2-44) >> >> Do you think we need the Sun java instead of the Red Hat supplied >> GNU java? That's what my sysadmin Tony suggests maybe? >> >> Using the instructions at: http://blacklight.rubyforge.org/files/DEMO_README.html >> . >> >> >> Jamie Orchard-Hays wrote: >> >>> OS: >>> Java version: >>> Specific instructions you're using: >>> Other significant data points: >>> >>> On Jun 18, 2009, at 11:55 AM, Jonathan Rochkind wrote: >>> >>> >>> >>>> Going through the demo instructions, trying to start solr with >>>> jetty as included with the demo app. Run into the following. Any >>>> ideas or advice would be appreciated. >>>> >>>> 18-Jun-09 11:53:22 AM org.apache.solr.servlet.SolrDispatchFilter >>>> init >>>> SEVERE: Could not start SOLR. Check solr/home property >>>> java.lang.NoClassDefFoundError: org.apache.solr.search.FastLRUCache >>>> >>>> [stack trace omitted] >>>> >>>> and... >>>> >>>> 2009-06-18 11:53:22.681::WARN: failed SolrRequestFilter >>>> java.lang.NoClassDefFoundError: org.apache.solr.core.SolrCore >>>> >>>> [stack trace omitted] >>>> >>>> >>>> [And some more, but I'm guessing they're caused by the first >>>> ones, perhaps all caused by the missing FastLRUCache ? ] >>>> >>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > _______________________________________________ Blacklight-development mailing list Blacklight-development at rubyforge.org http://rubyforge.org/mailman/listinfo/blacklight-development Blacklightopac Blog http://blacklightopac.org/ From jamie at dang.com Thu Jun 18 18:51:26 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Thu, 18 Jun 2009 18:51:26 -0400 Subject: [Blacklight-development] problems starting solr from 'demo' app In-Reply-To: <2AC333D68D50E241AFA6FF41ACD5E65A12B255F5D7@JHEMTEXVS1.win.ad.jhu.edu> References: <4A3A637B.3010404@jhu.edu> <4B3B9878-DF22-4438-8C63-6D565358BC96@dang.com> <4A3A7C3F.9020605@jhu.edu> <018B1AFD-630B-4593-B74A-9195604A3B77@mac.com> <4A3AA3FD.3000408@jhu.edu> <2AC333D68D50E241AFA6FF41ACD5E65A12B255F5D7@JHEMTEXVS1.win.ad.jhu.edu> Message-ID: <998ECBAF-4D96-4FF9-AFBA-552222492D69@dang.com> I wouldn't use ruby 1.8.7 anyway. :) Once you get RubyGems up, you have to get up to 1.3.x using it's self update feature. I highly recommend Phusion's Enterprise Ruby and Passenger for running your Rails apps, including blacklight. Only thing is, you'll probably still need a genera ruby installation, because you're using Capistrano to deploy (right? :). Jamie On Jun 18, 2009, at 5:07 PM, Tony Placilla wrote: >> From the FWIW dept. > > As Jonathan mentioned I am *not* a fan of using code compiled & > installed from source. And being as we're a RedHat shop I need it to > play nice with RPM for package management. > > Java was a piece of cake. > yum install java-1.5.0-sun.x86_64 > > & Bjorn Stronginthearm's your uncle! > > The Ruby issue was a bit more complex (but not too) & it remains to > be seen if the version we've got installed will provide all the > tools needed. > > 1.8.7 doesn't exist in an RH repository other than Rawhide. Thank > you, but no. > > Fedora 10, however, has ruby-1.8.6 in an SRPM > wget ftp://ftp.pbone.net/mirror/download.fedora.redhat.com/pub/fedora/linux/releases/10/Everything/source/SRPMS/ruby-1.8.6.287-2.fc10.src.rpm > > yum install tcl-devel tk-devel libX11-devel > rpmbuild --rebuild ruby-1.8.6.287-2.fc10.src.rpm > > cd /usr/src/redhat/RPMS/x86_64/ & install the binaries (assuming of > course that you have removed the RHEL supplied ones) > > Then > wget ftp://ftp.pbone.net/mirror/download.fedora.redhat.com/pub/fedora/linux/releases/10/Everything/source/SRPMS/rubygems-1.2.0-2.fc10.src.rpm > > lather, rinse, repeat as above > > > Tony Placilla > Sr. UNIX Systems Administrator > The Sheridan Libraries > Johns Hopkins University From rochkind at jhu.edu Fri Jun 19 11:42:09 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Fri, 19 Jun 2009 11:42:09 -0400 Subject: [Blacklight-development] advice for jetty setup? -- deploy a prototype app help Message-ID: <4A3BB1D1.3000603@jhu.edu> So I want to get a bare bones blacklight up for others on the implementation team to look at. I know that some of you all have such publically-available blacklights up, I appreciate any advice in the easiest way to make that so. I'm used to using mongrel for deployment, so figured I'd do that for now, even if it's not what we end up with for production deployment. Curious if anyone else is using mongrel, or else what you are using, are you all using passenger for any public-facing deployment? Oddly, I'm having a bit of trouble with mongrel and the --prefix argument I'm used to using to deploy a Rails app at a web path other than root. For some reason it's raising an error with the demo app, have to google around for that. Then I have questions about jetty, which I'm not used to. Are any of you guys in fact using jetty for deploying a publically-available version of blacklight? (Also, am I right that for an ultimate production deployment, jetty would not be reccommended, it's really just for testing?) Using the demo app, can anyone give me some advice for ensuring that jetty automatically starts on machine bootup (or when the rails app is started?) Thanks a lot for any advice. I will happily write up some documentation (on the wiki?) for others coming along who might have the same question. Jonathan From jamie at dang.com Fri Jun 19 12:04:41 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Fri, 19 Jun 2009 12:04:41 -0400 Subject: [Blacklight-development] advice for jetty setup? -- deploy a prototype app help In-Reply-To: <4A3BB1D1.3000603@jhu.edu> References: <4A3BB1D1.3000603@jhu.edu> Message-ID: Jonathan, for my rails deployments, I've been using the RailsMachine gem along with Capistrano for a while now. It makes certain assumptions and does certain things for you. (They've taken it to a new level with Moonshine, but it assumes Ubuntu 8.10 as your target.) https://support.railsmachine.com/index.php?pg=kb.chapter&id=18 RM gem assumes these defaults (but you can change them): the deploy user is "deploy" the root path for your apps is /var/www/apps/ You can configure Apache's virtual host for your project right from your capistrano deploy file. It's really easy. Have a look at their docs. If you install the gem then "capify ." your rails app, you get a deploy.rb file with a bunch of info regarding the RM additions. Oh, it's setup to work with both mongrel and passenger deployments. I highly recommend using passenger. It's easy as pie to install. The RailsMachine gem will handle the config stuff for you (cap web:setup). WRT Jetty, it's a very capable production server. Fast, light, etc. It doesn't come with start/stop scripts like Tomcat does. I'm not a big help there as my current projects with java web servers use tomcat. On Jun 19, 2009, at 11:42 AM, Jonathan Rochkind wrote: > So I want to get a bare bones blacklight up for others on the > implementation team to look at. I know that some of you all have > such publically-available blacklights up, I appreciate any advice in > the easiest way to make that so. > > I'm used to using mongrel for deployment, so figured I'd do that for > now, even if it's not what we end up with for production deployment. > Curious if anyone else is using mongrel, or else what you are using, > are you all using passenger for any public-facing deployment? > > Oddly, I'm having a bit of trouble with mongrel and the --prefix > argument I'm used to using to deploy a Rails app at a web path other > than root. For some reason it's raising an error with the demo app, > have to google around for that. > > Then I have questions about jetty, which I'm not used to. Are any of > you guys in fact using jetty for deploying a publically-available > version of blacklight? (Also, am I right that for an ultimate > production deployment, jetty would not be reccommended, it's really > just for testing?) > > > Using the demo app, can anyone give me some advice for ensuring that > jetty automatically starts on machine bootup (or when the rails app is > started?) > > Thanks a lot for any advice. I will happily write up some > documentation > (on the wiki?) for others coming along who might have the same > question. > > > Jonathan > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Fri Jun 19 12:24:20 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Fri, 19 Jun 2009 12:24:20 -0400 Subject: [Blacklight-development] advice for jetty setup? -- deploy a prototype app help In-Reply-To: References: <4A3BB1D1.3000603@jhu.edu> Message-ID: <4A3BBBB4.2060209@jhu.edu> Thanks. I'll take a look at RailsMachine and Capistrano. Are you guys using exlusively passenger for all public facing deployments, or are you using mongrel sometimes too? So maybe I got the wrong idea thinking Jetty wasn't suitable for production deployment, you guys think it is? Either way, still hoping for help on how to get a Jetty server with Solr to start up automatically, either when the Rails app starts (wonder if capistrano could somehow take care of that?), or on server boot, or whatever. Jamie Orchard-Hays wrote: > Jonathan, for my rails deployments, I've been using the RailsMachine > gem along with Capistrano for a while now. It makes certain > assumptions and does certain things for you. (They've taken it to a > new level with Moonshine, but it assumes Ubuntu 8.10 as your target.) > > https://support.railsmachine.com/index.php?pg=kb.chapter&id=18 > > RM gem assumes these defaults (but you can change them): > the deploy user is "deploy" > the root path for your apps is /var/www/apps/ > > You can configure Apache's virtual host for your project right from > your capistrano deploy file. It's really easy. > > Have a look at their docs. If you install the gem then "capify ." your > rails app, you get a deploy.rb file with a bunch of info regarding the > RM additions. > > Oh, it's setup to work with both mongrel and passenger deployments. I > highly recommend using passenger. It's easy as pie to install. The > RailsMachine gem will handle the config stuff for you (cap web:setup). > > WRT Jetty, it's a very capable production server. Fast, light, etc. It > doesn't come with start/stop scripts like Tomcat does. I'm not a big > help there as my current projects with java web servers use tomcat. > > > > > On Jun 19, 2009, at 11:42 AM, Jonathan Rochkind wrote: > > >> So I want to get a bare bones blacklight up for others on the >> implementation team to look at. I know that some of you all have >> such publically-available blacklights up, I appreciate any advice in >> the easiest way to make that so. >> >> I'm used to using mongrel for deployment, so figured I'd do that for >> now, even if it's not what we end up with for production deployment. >> Curious if anyone else is using mongrel, or else what you are using, >> are you all using passenger for any public-facing deployment? >> >> Oddly, I'm having a bit of trouble with mongrel and the --prefix >> argument I'm used to using to deploy a Rails app at a web path other >> than root. For some reason it's raising an error with the demo app, >> have to google around for that. >> >> Then I have questions about jetty, which I'm not used to. Are any of >> you guys in fact using jetty for deploying a publically-available >> version of blacklight? (Also, am I right that for an ultimate >> production deployment, jetty would not be reccommended, it's really >> just for testing?) >> >> >> Using the demo app, can anyone give me some advice for ensuring that >> jetty automatically starts on machine bootup (or when the rails app is >> started?) >> >> Thanks a lot for any advice. I will happily write up some >> documentation >> (on the wiki?) for others coming along who might have the same >> question. >> >> >> Jonathan >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From rochkind at jhu.edu Fri Jun 19 14:22:00 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Fri, 19 Jun 2009 14:22:00 -0400 Subject: [Blacklight-development] advice for jetty setup? -- deploy a prototype app help In-Reply-To: <23b83f160906191114g39486d49x927fc0d57cc07f2@mail.gmail.com> References: <4A3BB1D1.3000603@jhu.edu> <4A3BBBB4.2060209@jhu.edu> <23b83f160906191114g39486d49x927fc0d57cc07f2@mail.gmail.com> Message-ID: <4A3BD748.1000401@jhu.edu> Thanks Ross. Yeah, it's gonna be a completely different server. But really, ultimate production architecture can wait until later, for now I just need to get it visible by someone other than me, for demo/prototyping. Actual production is a ways off. So, off to work on figuring out how to start and stop jetty running as a daemon, and have it start on boot. Thanks everyone. Ross Singer wrote: > Jetty is certainly capable for a production system (we use it for the Platform, for example). I'd wager there's a ton of production Solrs-in-Jetty out there (not to mention a ton of other miscellaneous-servlets-in-Jetty). > > But, realistically, it's another service you need to look after. Since you seem to already have Tomcat in use at Hopkins, maybe you should deploy it in there? > > If this is going to be running on a completely different server, then I'd just go with Jetty. > > -Ross. > > On Fri, Jun 19, 2009 at 12:24 PM, Jonathan Rochkind > wrote: > Thanks. I'll take a look at RailsMachine and Capistrano. Are you guys using exlusively passenger for all public facing deployments, or are you using mongrel sometimes too? > > So maybe I got the wrong idea thinking Jetty wasn't suitable for production deployment, you guys think it is? Either way, still hoping for help on how to get a Jetty server with Solr to start up automatically, either when the Rails app starts (wonder if capistrano could somehow take care of that?), or on server boot, or whatever. > > > > Jamie Orchard-Hays wrote: > Jonathan, for my rails deployments, I've been using the RailsMachine gem along with Capistrano for a while now. It makes certain assumptions and does certain things for you. (They've taken it to a new level with Moonshine, but it assumes Ubuntu 8.10 as your target.) > > https://support.railsmachine.com/index.php?pg=kb.chapter&id=18 > > RM gem assumes these defaults (but you can change them): > the deploy user is "deploy" > the root path for your apps is /var/www/apps/ > > You can configure Apache's virtual host for your project right from your capistrano deploy file. It's really easy. > > Have a look at their docs. If you install the gem then "capify ." your rails app, you get a deploy.rb file with a bunch of info regarding the RM additions. > > Oh, it's setup to work with both mongrel and passenger deployments. I highly recommend using passenger. It's easy as pie to install. The RailsMachine gem will handle the config stuff for you (cap web:setup). > > WRT Jetty, it's a very capable production server. Fast, light, etc. It doesn't come with start/stop scripts like Tomcat does. I'm not a big help there as my current projects with java web servers use tomcat. > > > > > On Jun 19, 2009, at 11:42 AM, Jonathan Rochkind wrote: > > > So I want to get a bare bones blacklight up for others on the implementation team to look at. I know that some of you all have such publically-available blacklights up, I appreciate any advice in the easiest way to make that so. > > I'm used to using mongrel for deployment, so figured I'd do that for now, even if it's not what we end up with for production deployment. Curious if anyone else is using mongrel, or else what you are using, are you all using passenger for any public-facing deployment? > > Oddly, I'm having a bit of trouble with mongrel and the --prefix argument I'm used to using to deploy a Rails app at a web path other than root. For some reason it's raising an error with the demo app, have to google around for that. > > Then I have questions about jetty, which I'm not used to. Are any of you guys in fact using jetty for deploying a publically-available version of blacklight? (Also, am I right that for an ultimate production deployment, jetty would not be reccommended, it's really just for testing?) > > > Using the demo app, can anyone give me some advice for ensuring that > jetty automatically starts on machine bootup (or when the rails app is > started?) > > Thanks a lot for any advice. I will happily write up some documentation > (on the wiki?) for others coming along who might have the same question. > > > Jonathan > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > From eos8d at virginia.edu Mon Jun 22 11:14:43 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Mon, 22 Jun 2009 11:14:43 -0400 Subject: [Blacklight-development] advice for jetty setup? -- deploy a prototype app help In-Reply-To: <4A3BD748.1000401@jhu.edu> References: <4A3BB1D1.3000603@jhu.edu> <4A3BBBB4.2060209@jhu.edu> <23b83f160906191114g39486d49x927fc0d57cc07f2@mail.gmail.com> <4A3BD748.1000401@jhu.edu> Message-ID: <37E09F98-CE3D-48E1-9F47-6506117C5765@virginia.edu> On 19-Jun-09, at 2:22 PM, Jonathan Rochkind wrote: > So, off to work on figuring out how to start and stop jetty running > as a > daemon, and have it start on boot. Thanks everyone. Jonathan, I know you say production implementation can wait for awhile, but if the system you're setting up is far enough into production that you want solr to run as a service, I strongly suggest you go with tomcat instead. Tomcat6 runs beautifully as a red hat package (we install via jpackage, and I can send the install docs along if that's of interest). It's only slightly more difficult to install than out-of-the-box jetty, but IMHO it's a better time investment than putting effort into running jetty as a daemon, with startup and shutdown scripts, if jetty isn't what you're ultimately planning to use. Let me know if you'd like me to share our setup and installation docs for this. Bess From eos8d at virginia.edu Mon Jun 22 11:19:37 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Mon, 22 Jun 2009 11:19:37 -0400 Subject: [Blacklight-development] advice for jetty setup? -- deploy a prototype app help In-Reply-To: <37E09F98-CE3D-48E1-9F47-6506117C5765@virginia.edu> References: <4A3BB1D1.3000603@jhu.edu> <4A3BBBB4.2060209@jhu.edu> <23b83f160906191114g39486d49x927fc0d57cc07f2@mail.gmail.com> <4A3BD748.1000401@jhu.edu> <37E09F98-CE3D-48E1-9F47-6506117C5765@virginia.edu> Message-ID: <032F6562-D5A5-4CCC-B5DA-55C55FFF1165@virginia.edu> And now that I read back over the rest of the thread (sorry, just catching up) I'd strongly recommend you go ahead and use passenger too, for anything but the most bare-bones, this is my personal copy that I run on my laptop installation. It isn't at all difficult to set up, it gives you loads of advantages, and you do occasionally run into slightly different behaviors b/w mongrel and passenger. I wouldn't want to be developing on mongrel and then deploying to passenger, because I wouldn't feel like the app had been properly tested. FYI, I run passenger even on my laptop for local development. I love it. Bess On 22-Jun-09, at 11:14 AM, Bess Sadler wrote: > > On 19-Jun-09, at 2:22 PM, Jonathan Rochkind wrote: > >> So, off to work on figuring out how to start and stop jetty running >> as a >> daemon, and have it start on boot. Thanks everyone. > > Jonathan, I know you say production implementation can wait for > awhile, but if the system you're setting up is far enough into > production that you want solr to run as a service, I strongly suggest > you go with tomcat instead. Tomcat6 runs beautifully as a red hat > package (we install via jpackage, and I can send the install docs > along if that's of interest). It's only slightly more difficult to > install than out-of-the-box jetty, but IMHO it's a better time > investment than putting effort into running jetty as a daemon, with > startup and shutdown scripts, if jetty isn't what you're ultimately > planning to use. > > Let me know if you'd like me to share our setup and installation docs > for this. > > Bess > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Mon Jun 22 11:21:01 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Mon, 22 Jun 2009 11:21:01 -0400 Subject: [Blacklight-development] advice for jetty setup? -- deploy a prototype app help In-Reply-To: <37E09F98-CE3D-48E1-9F47-6506117C5765@virginia.edu> References: <4A3BB1D1.3000603@jhu.edu> <4A3BBBB4.2060209@jhu.edu> <23b83f160906191114g39486d49x927fc0d57cc07f2@mail.gmail.com> <4A3BD748.1000401@jhu.edu> <37E09F98-CE3D-48E1-9F47-6506117C5765@virginia.edu> Message-ID: <4A3FA15D.2010000@jhu.edu> Thanks Bess. From the feedback I got on the list, I think Jetty probably _is_ what we're ultimately planning on using -- people seem to agree that jetty is capable of production, this will end up on a VM that won't already have tomcat installed, and my sysadmin prefers jetty to tomcat. Is either UVa or Stanford using jetty in production for deploying solr, or are you all using tomcat? Bess Sadler wrote: > On 19-Jun-09, at 2:22 PM, Jonathan Rochkind wrote: > > >> So, off to work on figuring out how to start and stop jetty running >> as a >> daemon, and have it start on boot. Thanks everyone. >> > > Jonathan, I know you say production implementation can wait for > awhile, but if the system you're setting up is far enough into > production that you want solr to run as a service, I strongly suggest > you go with tomcat instead. Tomcat6 runs beautifully as a red hat > package (we install via jpackage, and I can send the install docs > along if that's of interest). It's only slightly more difficult to > install than out-of-the-box jetty, but IMHO it's a better time > investment than putting effort into running jetty as a daemon, with > startup and shutdown scripts, if jetty isn't what you're ultimately > planning to use. > > Let me know if you'd like me to share our setup and installation docs > for this. > > Bess > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From eos8d at virginia.edu Mon Jun 22 11:25:42 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Mon, 22 Jun 2009 11:25:42 -0400 Subject: [Blacklight-development] advice for jetty setup? -- deploy a prototype app help In-Reply-To: <4A3FA15D.2010000@jhu.edu> References: <4A3BB1D1.3000603@jhu.edu> <4A3BBBB4.2060209@jhu.edu> <23b83f160906191114g39486d49x927fc0d57cc07f2@mail.gmail.com> <4A3BD748.1000401@jhu.edu> <37E09F98-CE3D-48E1-9F47-6506117C5765@virginia.edu> <4A3FA15D.2010000@jhu.edu> Message-ID: <01B70155-A4E7-4FD2-963B-A07E225BB3E0@virginia.edu> UVA is using tomcat 6. We've been very happy with it. We use jetty for local testing only. From what I hear jetty is fine for production, but we have much more local expertise running tomcat in a production environment, so that's what we go with. Bess On 22-Jun-09, at 11:21 AM, Jonathan Rochkind wrote: > Thanks Bess. From the feedback I got on the list, I think Jetty > probably _is_ what we're ultimately planning on using -- people seem > to > agree that jetty is capable of production, this will end up on a VM > that > won't already have tomcat installed, and my sysadmin prefers jetty to > tomcat. > > Is either UVa or Stanford using jetty in production for deploying > solr, > or are you all using tomcat? > > Bess Sadler wrote: >> On 19-Jun-09, at 2:22 PM, Jonathan Rochkind wrote: >> >> >>> So, off to work on figuring out how to start and stop jetty running >>> as a >>> daemon, and have it start on boot. Thanks everyone. >>> >> >> Jonathan, I know you say production implementation can wait for >> awhile, but if the system you're setting up is far enough into >> production that you want solr to run as a service, I strongly suggest >> you go with tomcat instead. Tomcat6 runs beautifully as a red hat >> package (we install via jpackage, and I can send the install docs >> along if that's of interest). It's only slightly more difficult to >> install than out-of-the-box jetty, but IMHO it's a better time >> investment than putting effort into running jetty as a daemon, with >> startup and shutdown scripts, if jetty isn't what you're ultimately >> planning to use. >> >> Let me know if you'd like me to share our setup and installation docs >> for this. >> >> Bess >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From ndushay at stanford.edu Mon Jun 22 12:22:43 2009 From: ndushay at stanford.edu (Naomi Dushay) Date: Mon, 22 Jun 2009 09:22:43 -0700 Subject: [Blacklight-development] advice for jetty setup? -- deploy a prototype app help In-Reply-To: <4A3FA15D.2010000@jhu.edu> References: <4A3BB1D1.3000603@jhu.edu> <4A3BBBB4.2060209@jhu.edu> <23b83f160906191114g39486d49x927fc0d57cc07f2@mail.gmail.com> <4A3BD748.1000401@jhu.edu> <37E09F98-CE3D-48E1-9F47-6506117C5765@virginia.edu> <4A3FA15D.2010000@jhu.edu> Message-ID: <02AF3F9D-21C7-4538-BD4F-F76015482483@stanford.edu> Stanford uses Jetty for SOLR, and it's scaled just fine for us so far. We will be using passenger for blacklight RoR, but we've been using Mongrel for now. - Naomi On Jun 22, 2009, at 8:21 AM, Jonathan Rochkind wrote: > Thanks Bess. From the feedback I got on the list, I think Jetty > probably _is_ what we're ultimately planning on using -- people seem > to agree that jetty is capable of production, this will end up on a > VM that won't already have tomcat installed, and my sysadmin prefers > jetty to tomcat. > > Is either UVa or Stanford using jetty in production for deploying > solr, or are you all using tomcat? > > Bess Sadler wrote: >> On 19-Jun-09, at 2:22 PM, Jonathan Rochkind wrote: >> >> >>> So, off to work on figuring out how to start and stop jetty >>> running as a >>> daemon, and have it start on boot. Thanks everyone. >>> >> >> Jonathan, I know you say production implementation can wait for >> awhile, but if the system you're setting up is far enough into >> production that you want solr to run as a service, I strongly >> suggest you go with tomcat instead. Tomcat6 runs beautifully as a >> red hat package (we install via jpackage, and I can send the >> install docs along if that's of interest). It's only slightly more >> difficult to install than out-of-the-box jetty, but IMHO it's a >> better time investment than putting effort into running jetty as a >> daemon, with startup and shutdown scripts, if jetty isn't what >> you're ultimately planning to use. >> >> Let me know if you'd like me to share our setup and installation >> docs for this. >> >> Bess >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Mon Jun 22 12:55:54 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Mon, 22 Jun 2009 12:55:54 -0400 Subject: [Blacklight-development] advice for jetty setup? -- deploy a prototype app help In-Reply-To: <032F6562-D5A5-4CCC-B5DA-55C55FFF1165@virginia.edu> References: <4A3BB1D1.3000603@jhu.edu> <4A3BBBB4.2060209@jhu.edu> <23b83f160906191114g39486d49x927fc0d57cc07f2@mail.gmail.com> <4A3BD748.1000401@jhu.edu> <37E09F98-CE3D-48E1-9F47-6506117C5765@virginia.edu> <032F6562-D5A5-4CCC-B5DA-55C55FFF1165@virginia.edu> Message-ID: <4A3FB79A.7060305@jhu.edu> Thanks for this too. To clarify, are you guys deploying in production with passenger? Or with... what? I thought I remembered you saying you weren't deploying in production with passenger, which, if you're using mongrel, means that we need to ensure BL works in both cases anyway, right? I do want to use passenger. My (non-technical) problem is I'm being told by supervisors that I need to have something I can 'show off' as a live demo/prototype asap with the emphasis on the 's', and the more tools that I'm unfamiliar with I add to my requirements list, the longer it's going to take me to have that. Bess Sadler wrote: > And now that I read back over the rest of the thread (sorry, just > catching up) I'd strongly recommend you go ahead and use passenger > too, for anything but the most bare-bones, this is my personal copy > that I run on my laptop installation. It isn't at all difficult to set > up, it gives you loads of advantages, and you do occasionally run into > slightly different behaviors b/w mongrel and passenger. I wouldn't > want to be developing on mongrel and then deploying to passenger, > because I wouldn't feel like the app had been properly tested. FYI, I > run passenger even on my laptop for local development. I love it. > > Bess > > On 22-Jun-09, at 11:14 AM, Bess Sadler wrote: > > >> On 19-Jun-09, at 2:22 PM, Jonathan Rochkind wrote: >> >> >>> So, off to work on figuring out how to start and stop jetty running >>> as a >>> daemon, and have it start on boot. Thanks everyone. >>> >> Jonathan, I know you say production implementation can wait for >> awhile, but if the system you're setting up is far enough into >> production that you want solr to run as a service, I strongly suggest >> you go with tomcat instead. Tomcat6 runs beautifully as a red hat >> package (we install via jpackage, and I can send the install docs >> along if that's of interest). It's only slightly more difficult to >> install than out-of-the-box jetty, but IMHO it's a better time >> investment than putting effort into running jetty as a daemon, with >> startup and shutdown scripts, if jetty isn't what you're ultimately >> planning to use. >> >> Let me know if you'd like me to share our setup and installation docs >> for this. >> >> Bess >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From jamie at dang.com Mon Jun 22 14:29:05 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Mon, 22 Jun 2009 14:29:05 -0400 Subject: [Blacklight-development] advice for jetty setup? -- deploy a prototype app help In-Reply-To: <4A3FB79A.7060305@jhu.edu> References: <4A3BB1D1.3000603@jhu.edu> <4A3BBBB4.2060209@jhu.edu> <23b83f160906191114g39486d49x927fc0d57cc07f2@mail.gmail.com> <4A3BD748.1000401@jhu.edu> <37E09F98-CE3D-48E1-9F47-6506117C5765@virginia.edu> <032F6562-D5A5-4CCC-B5DA-55C55FFF1165@virginia.edu> <4A3FB79A.7060305@jhu.edu> Message-ID: <4EAA5E52-5440-4553-A4F3-DC5A0B64316E@dang.com> Production and development. I moved all my clients' servers to Passenger. Seriously, it's not quite, but almost trivial to install. On Jun 22, 2009, at 12:55 PM, Jonathan Rochkind wrote: > Thanks for this too. > > To clarify, are you guys deploying in production with passenger? Or > with... what? I thought I remembered you saying you weren't > deploying in production with passenger, which, if you're using > mongrel, means that we need to ensure BL works in both cases anyway, > right? > > I do want to use passenger. My (non-technical) problem is I'm being > told by supervisors that I need to have something I can 'show off' > as a live demo/prototype asap with the emphasis on the 's', and the > more tools that I'm unfamiliar with I add to my requirements list, > the longer it's going to take me to have that. > > Bess Sadler wrote: >> And now that I read back over the rest of the thread (sorry, just >> catching up) I'd strongly recommend you go ahead and use passenger >> too, for anything but the most bare-bones, this is my personal >> copy that I run on my laptop installation. It isn't at all >> difficult to set up, it gives you loads of advantages, and you do >> occasionally run into slightly different behaviors b/w mongrel and >> passenger. I wouldn't want to be developing on mongrel and then >> deploying to passenger, because I wouldn't feel like the app had >> been properly tested. FYI, I run passenger even on my laptop for >> local development. I love it. >> >> Bess >> >> On 22-Jun-09, at 11:14 AM, Bess Sadler wrote: >> >> >>> On 19-Jun-09, at 2:22 PM, Jonathan Rochkind wrote: >>> >>> >>>> So, off to work on figuring out how to start and stop jetty running >>>> as a >>>> daemon, and have it start on boot. Thanks everyone. >>>> >>> Jonathan, I know you say production implementation can wait for >>> awhile, but if the system you're setting up is far enough into >>> production that you want solr to run as a service, I strongly >>> suggest >>> you go with tomcat instead. Tomcat6 runs beautifully as a red hat >>> package (we install via jpackage, and I can send the install docs >>> along if that's of interest). It's only slightly more difficult to >>> install than out-of-the-box jetty, but IMHO it's a better time >>> investment than putting effort into running jetty as a daemon, with >>> startup and shutdown scripts, if jetty isn't what you're ultimately >>> planning to use. >>> >>> Let me know if you'd like me to share our setup and installation >>> docs >>> for this. >>> >>> Bess >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From aplacilla at jhu.edu Mon Jun 22 14:32:14 2009 From: aplacilla at jhu.edu (Tony Placilla) Date: Mon, 22 Jun 2009 14:32:14 -0400 Subject: [Blacklight-development] advice for jetty setup? -- deploy a prototype app help In-Reply-To: <4EAA5E52-5440-4553-A4F3-DC5A0B64316E@dang.com> References: <4A3BB1D1.3000603@jhu.edu> <4A3BBBB4.2060209@jhu.edu> <23b83f160906191114g39486d49x927fc0d57cc07f2@mail.gmail.com> <4A3BD748.1000401@jhu.edu> <37E09F98-CE3D-48E1-9F47-6506117C5765@virginia.edu> <032F6562-D5A5-4CCC-B5DA-55C55FFF1165@virginia.edu> <4A3FB79A.7060305@jhu.edu> <4EAA5E52-5440-4553-A4F3-DC5A0B64316E@dang.com> Message-ID: <2AC333D68D50E241AFA6FF41ACD5E65A12B25EA211@JHEMTEXVS1.win.ad.jhu.edu> You mean "almost, but not quite, entirely unlike tea" :-) Tony Placilla Sr. UNIX Systems Administrator The Sheridan Libraries Johns Hopkins University -----Original Message----- From: blacklight-development-bounces at rubyforge.org [mailto:blacklight-development-bounces at rubyforge.org] On Behalf Of Jamie Orchard-Hays Sent: Monday, June 22, 2009 2:29 PM To: blacklight-development at rubyforge.org Subject: Re: [Blacklight-development] advice for jetty setup? -- deploy a prototype app help Production and development. I moved all my clients' servers to Passenger. Seriously, it's not quite, but almost trivial to install. On Jun 22, 2009, at 12:55 PM, Jonathan Rochkind wrote: > Thanks for this too. > > To clarify, are you guys deploying in production with passenger? Or > with... what? I thought I remembered you saying you weren't > deploying in production with passenger, which, if you're using > mongrel, means that we need to ensure BL works in both cases anyway, > right? > > I do want to use passenger. My (non-technical) problem is I'm being > told by supervisors that I need to have something I can 'show off' > as a live demo/prototype asap with the emphasis on the 's', and the > more tools that I'm unfamiliar with I add to my requirements list, > the longer it's going to take me to have that. > > Bess Sadler wrote: >> And now that I read back over the rest of the thread (sorry, just >> catching up) I'd strongly recommend you go ahead and use passenger >> too, for anything but the most bare-bones, this is my personal >> copy that I run on my laptop installation. It isn't at all >> difficult to set up, it gives you loads of advantages, and you do >> occasionally run into slightly different behaviors b/w mongrel and >> passenger. I wouldn't want to be developing on mongrel and then >> deploying to passenger, because I wouldn't feel like the app had >> been properly tested. FYI, I run passenger even on my laptop for >> local development. I love it. >> >> Bess >> >> On 22-Jun-09, at 11:14 AM, Bess Sadler wrote: >> >> >>> On 19-Jun-09, at 2:22 PM, Jonathan Rochkind wrote: >>> >>> >>>> So, off to work on figuring out how to start and stop jetty running >>>> as a >>>> daemon, and have it start on boot. Thanks everyone. >>>> >>> Jonathan, I know you say production implementation can wait for >>> awhile, but if the system you're setting up is far enough into >>> production that you want solr to run as a service, I strongly >>> suggest >>> you go with tomcat instead. Tomcat6 runs beautifully as a red hat >>> package (we install via jpackage, and I can send the install docs >>> along if that's of interest). It's only slightly more difficult to >>> install than out-of-the-box jetty, but IMHO it's a better time >>> investment than putting effort into running jetty as a daemon, with >>> startup and shutdown scripts, if jetty isn't what you're ultimately >>> planning to use. >>> >>> Let me know if you'd like me to share our setup and installation >>> docs >>> for this. >>> >>> Bess >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ _______________________________________________ Blacklight-development mailing list Blacklight-development at rubyforge.org http://rubyforge.org/mailman/listinfo/blacklight-development Blacklightopac Blog http://blacklightopac.org/ From eos8d at virginia.edu Mon Jun 22 14:38:23 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Mon, 22 Jun 2009 14:38:23 -0400 Subject: [Blacklight-development] advice for jetty setup? -- deploy a prototype app help In-Reply-To: <4EAA5E52-5440-4553-A4F3-DC5A0B64316E@dang.com> References: <4A3BB1D1.3000603@jhu.edu> <4A3BBBB4.2060209@jhu.edu> <23b83f160906191114g39486d49x927fc0d57cc07f2@mail.gmail.com> <4A3BD748.1000401@jhu.edu> <37E09F98-CE3D-48E1-9F47-6506117C5765@virginia.edu> <032F6562-D5A5-4CCC-B5DA-55C55FFF1165@virginia.edu> <4A3FB79A.7060305@jhu.edu> <4EAA5E52-5440-4553-A4F3-DC5A0B64316E@dang.com> Message-ID: <665ED505-9B5F-42AB-90DE-BD189928E2AE@virginia.edu> UVA is using passenger for production exclusively, and I wouldn't use anything else for production. We found mongrel just crashes too often, for no discernible reason. Mongrel is fine for running on your laptop, but if it's anything for which you want even the smallest amount of stability, and if it's anything for which you need to install apache httpd as a front end anyway, install passenger. It's faster, easier to install and more stable. Bess On 22-Jun-09, at 2:29 PM, Jamie Orchard-Hays wrote: > Production and development. I moved all my clients' servers to > Passenger. > > Seriously, it's not quite, but almost trivial to install. > > > On Jun 22, 2009, at 12:55 PM, Jonathan Rochkind wrote: > >> Thanks for this too. >> >> To clarify, are you guys deploying in production with passenger? Or >> with... what? I thought I remembered you saying you weren't >> deploying in production with passenger, which, if you're using >> mongrel, means that we need to ensure BL works in both cases anyway, >> right? >> >> I do want to use passenger. My (non-technical) problem is I'm being >> told by supervisors that I need to have something I can 'show off' >> as a live demo/prototype asap with the emphasis on the 's', and the >> more tools that I'm unfamiliar with I add to my requirements list, >> the longer it's going to take me to have that. >> >> Bess Sadler wrote: >>> And now that I read back over the rest of the thread (sorry, just >>> catching up) I'd strongly recommend you go ahead and use passenger >>> too, for anything but the most bare-bones, this is my personal >>> copy that I run on my laptop installation. It isn't at all >>> difficult to set up, it gives you loads of advantages, and you do >>> occasionally run into slightly different behaviors b/w mongrel and >>> passenger. I wouldn't want to be developing on mongrel and then >>> deploying to passenger, because I wouldn't feel like the app had >>> been properly tested. FYI, I run passenger even on my laptop for >>> local development. I love it. >>> >>> Bess >>> >>> On 22-Jun-09, at 11:14 AM, Bess Sadler wrote: >>> >>> >>>> On 19-Jun-09, at 2:22 PM, Jonathan Rochkind wrote: >>>> >>>> >>>>> So, off to work on figuring out how to start and stop jetty >>>>> running >>>>> as a >>>>> daemon, and have it start on boot. Thanks everyone. >>>>> >>>> Jonathan, I know you say production implementation can wait for >>>> awhile, but if the system you're setting up is far enough into >>>> production that you want solr to run as a service, I strongly >>>> suggest >>>> you go with tomcat instead. Tomcat6 runs beautifully as a red hat >>>> package (we install via jpackage, and I can send the install docs >>>> along if that's of interest). It's only slightly more difficult to >>>> install than out-of-the-box jetty, but IMHO it's a better time >>>> investment than putting effort into running jetty as a daemon, with >>>> startup and shutdown scripts, if jetty isn't what you're ultimately >>>> planning to use. >>>> >>>> Let me know if you'd like me to share our setup and installation >>>> docs >>>> for this. >>>> >>>> Bess >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Mon Jun 22 16:44:54 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Mon, 22 Jun 2009 16:44:54 -0400 Subject: [Blacklight-development] error in marc indexing? Message-ID: <4A3FED46.10106@jhu.edu> Trying to take one baby step at a time in getting a little demo of Blacklight with our demo, I'm trying to index a sample of our own local MARC records. I get an error from "rake app:index:marc", not sure exactly why, error below. Maybe errors in my MARC? There definitely _will_ be illegalities in my marc, would rather the indexer recovered from them somehow (or just skipped that record, if nothing else is possible), rather than punted the entire input. Should I switch to SolrMARC instead, is it a bit more forgiving? At one point I know I saw a pointer in the documentation to SolrMarc docs, to help you get started with solrmarc instead of the bundled ruby indexer... but now I can't seem to find it. Here's what "raek app:index:marc" is telling me (giving me an html error message even though it's a command-line rake task, which is a bit odd). rake aborted! Error 400

HTTP ERROR: 400

ERROR: [53567] multiple values 
encountered for non multiValued field title_t: [Honan's Handbook to 
medical Europe, A ready reference book to the universities, hospitals, 
clinics, laboratories and general medical work of the principal cities 
of Europe]

RequestURI=/solr/update

Powered by Jetty://


From eos8d at virginia.edu Mon Jun 22 17:02:33 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Mon, 22 Jun 2009 17:02:33 -0400 Subject: [Blacklight-development] error in marc indexing? In-Reply-To: <4A3FED46.10106@jhu.edu> References: <4A3FED46.10106@jhu.edu> Message-ID: <5D09E360-C37D-4435-8E4E-84974474CAA4@virginia.edu> Hi, Jonathan. The latest download of the demo app should be using solrmarc for indexing, and solrmarc is pretty robust at handling badly formed marc records and recovering. Could you verify which version you're using? The first thing I want to verify is that you are indeed using a version that uses solrmarc for indexing. The reason you're getting an HTML error is because the error is being thrown by solr, not by the indexer itself. Are you using the solr config that came with blacklight? Or did you set your own up? Regardless, the problem seems to be that solr is set up with a field called title_t that isn't allowed to be multi-valued, but you're passing an array of values to it. One easy fix is to change your solr config to let title_t be multi-valued. Or you could change your index mapping to only grab the first title from any record. It's probably easier to adjust solr. In your solr_home/conf, look for the file called schema.xml. Find a line like this: and change it to a line like this: FYI, I'm not sure what the display will do when you start having multivalued titles. You may need to go through and override any methods that access the title field to either iterate through an array or grab the first value of an array instead of just grabbing the value of the field itself. If you're still having trouble, could you send the marc record in question to the list? Or to me off-list, if the mailing list doesn't accept attachments. Bess On 22-Jun-09, at 4:44 PM, Jonathan Rochkind wrote: > Trying to take one baby step at a time in getting a little demo of > Blacklight with our demo, I'm trying to index a sample of our own > local > MARC records. > > I get an error from "rake app:index:marc", not sure exactly why, error > below. Maybe errors in my MARC? There definitely _will_ be > illegalities > in my marc, would rather the indexer recovered from them somehow (or > just skipped that record, if nothing else is possible), rather than > punted the entire input. > > Should I switch to SolrMARC instead, is it a bit more forgiving? At > one > point I know I saw a pointer in the documentation to SolrMarc docs, to > help you get started with solrmarc instead of the bundled ruby > indexer... but now I can't seem to find it. > > Here's what "raek app:index:marc" is telling me (giving me an html > error > message even though it's a command-line rake task, which is a bit > odd). > > rake aborted! > > > > Error 400 > >

HTTP ERROR: 400

ERROR: [53567] multiple values
> encountered for non multiValued field title_t: [Honan's Handbook to
> medical Europe, A ready reference book to the universities, hospitals,
> clinics, laboratories and general medical work of the principal cities
> of Europe]
>

RequestURI=/solr/update

href="http://jetty.mortbay.org/">Powered by > Jetty://


> _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From ndushay at stanford.edu Mon Jun 22 17:13:44 2009 From: ndushay at stanford.edu (Naomi Dushay) Date: Mon, 22 Jun 2009 14:13:44 -0700 Subject: [Blacklight-development] error in marc indexing? In-Reply-To: <4A3FED46.10106@jhu.edu> References: <4A3FED46.10106@jhu.edu> Message-ID: Jonathan, We have a TON of bad data. Lots of records with multiple 245a; lots of records with other similar problems. We use solrmarc, and it nicely steps around these problems. There are solrmarc lists; solrmarc-general at googlegroups.com solrmarc-technical at googlegroups.com - Naomi On Jun 22, 2009, at 1:44 PM, Jonathan Rochkind wrote: > Trying to take one baby step at a time in getting a little demo of > Blacklight with our demo, I'm trying to index a sample of our own > local MARC records. > > I get an error from "rake app:index:marc", not sure exactly why, > error below. Maybe errors in my MARC? There definitely _will_ be > illegalities in my marc, would rather the indexer recovered from > them somehow (or just skipped that record, if nothing else is > possible), rather than punted the entire input. > > Should I switch to SolrMARC instead, is it a bit more forgiving? At > one point I know I saw a pointer in the documentation to SolrMarc > docs, to help you get started with solrmarc instead of the bundled > ruby indexer... but now I can't seem to find it. > > Here's what "raek app:index:marc" is telling me (giving me an html > error message even though it's a command-line rake task, which is a > bit odd). > rake aborted! > > > > Error 400 > >

HTTP ERROR: 400

ERROR: [53567] multiple values  
> encountered for non multiValued field title_t: [Honan's Handbook to  
> medical Europe, A ready reference book to the universities,  
> hospitals, clinics, laboratories and general medical work of the  
> principal cities of Europe]
>

RequestURI=/solr/update

Powered by Jetty://


> _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Mon Jun 22 17:17:18 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Mon, 22 Jun 2009 17:17:18 -0400 Subject: [Blacklight-development] error in marc indexing? In-Reply-To: <5D09E360-C37D-4435-8E4E-84974474CAA4@virginia.edu> References: <4A3FED46.10106@jhu.edu> <5D09E360-C37D-4435-8E4E-84974474CAA4@virginia.edu> Message-ID: <4A3FF4DE.4060705@jhu.edu> Aha, cool. So when I run "rake app:index:marc" from the demo app, it should be using SolrMarc? That's awesome. What's the easiest way to verify if that's true or not? I have trunk of the demo app checked out though, so I guess it should be true. I just 'svn up'd my demo checkout to be sure, there were a few changes I got. I am indeed using solr config and everything else that came with 'demo', haven't tried customizing anything yet. Yeah, puttering around in the various documentation, I hit upon that multi-value thing as a guess too. And indeed, in (checked out from demo) /bl-demo/jetty/solr/conf/schema.xml, I've got: None of the rest of you have run into a case in your marc where the indexer as configured by default comes up with more than one title for a marc record? That seems surprising to me, it seems like it should be a common case? It makes me think that the 'demo' schema.xml should allow multiple values in that field, but then you warn that the demo app may not be prepared to actually deal with that. This seems odd to me. Anyone hazard a guess as to what's going on there, why nobody but me has run into marc that gets translated as having more than one title? Let me try to look up the particular record it's choking on... It's actually a fairly simple record, I'll paste in psuedo-marc-text below. Looks like it has two 245$a's. Looks like it's a really old title, which probably means it was cataloged to rare books rules, instead of AACR2. Which might explain why I've run into this but nobody else has, if you don't have marc in your catalogs like this -- we have a LOT of it, actually. Still not sure what to _do_ about it, and if the demo app or blacklight solrmarc config should be patched in some way to deal with it? Happy to work on the code and submit patches if we can come up with the proper thing it _ought_ to be doing. Does seem to me like the standard out of the box stuff ought to be able to deal with a record like this. 005: 19970719131600.0 008: 830722s1912 pa ab 000 0 eng d 035: $a 53567 035: $a Cz% 035: $a 09727331 $c 860228 040: $a SBH $c SBH 049: $a JHWD $n o 090: $a R771 $b .H76 1912 100: 1 $a Honan, James Henry, $d 1859-1917. 245: 10 $a Honan's Handbook to medical Europe. $a A ready reference book to the universities, hospitals, clinics, laboratories and general medical work of the principal cities of Europe, $c by James Henry Honan. 260: $a Philadelphia: $b P. Blakiston, $c 1912. 300: $a viii, 261 p. $b maps, tables. 500: $a Association : Matthew D. Mann. 740: 0 $a Handbook to medical Europe. 910: $a 53567 $b Horizon bib# Bess Sadler wrote: > Hi, Jonathan. > > The latest download of the demo app should be using solrmarc for > indexing, and solrmarc is pretty robust at handling badly formed marc > records and recovering. Could you verify which version you're using? > The first thing I want to verify is that you are indeed using a > version that uses solrmarc for indexing. > > The reason you're getting an HTML error is because the error is being > thrown by solr, not by the indexer itself. Are you using the solr > config that came with blacklight? Or did you set your own up? > Regardless, the problem seems to be that solr is set up with a field > called title_t that isn't allowed to be multi-valued, but you're > passing an array of values to it. One easy fix is to change your solr > config to let title_t be multi-valued. Or you could change your index > mapping to only grab the first title from any record. It's probably > easier to adjust solr. > > In your solr_home/conf, look for the file called schema.xml. Find a > line like this: > > > > and change it to a line like this: > > multiValued="true" /> > > FYI, I'm not sure what the display will do when you start having > multivalued titles. You may need to go through and override any > methods that access the title field to either iterate through an array > or grab the first value of an array instead of just grabbing the value > of the field itself. > > If you're still having trouble, could you send the marc record in > question to the list? Or to me off-list, if the mailing list doesn't > accept attachments. > > Bess > > > On 22-Jun-09, at 4:44 PM, Jonathan Rochkind wrote: > > >> Trying to take one baby step at a time in getting a little demo of >> Blacklight with our demo, I'm trying to index a sample of our own >> local >> MARC records. >> >> I get an error from "rake app:index:marc", not sure exactly why, error >> below. Maybe errors in my MARC? There definitely _will_ be >> illegalities >> in my marc, would rather the indexer recovered from them somehow (or >> just skipped that record, if nothing else is possible), rather than >> punted the entire input. >> >> Should I switch to SolrMARC instead, is it a bit more forgiving? At >> one >> point I know I saw a pointer in the documentation to SolrMarc docs, to >> help you get started with solrmarc instead of the bundled ruby >> indexer... but now I can't seem to find it. >> >> Here's what "raek app:index:marc" is telling me (giving me an html >> error >> message even though it's a command-line rake task, which is a bit >> odd). >> >> rake aborted! >> >> >> >> Error 400 >> >>

HTTP ERROR: 400

ERROR: [53567] multiple values
>> encountered for non multiValued field title_t: [Honan's Handbook to
>> medical Europe, A ready reference book to the universities, hospitals,
>> clinics, laboratories and general medical work of the principal cities
>> of Europe]
>>

RequestURI=/solr/update

> href="http://jetty.mortbay.org/">Powered by >> Jetty://


>> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From rochkind at jhu.edu Mon Jun 22 17:20:30 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Mon, 22 Jun 2009 17:20:30 -0400 Subject: [Blacklight-development] error in marc indexing? In-Reply-To: References: <4A3FED46.10106@jhu.edu> Message-ID: <4A3FF59E.4030803@jhu.edu> Cool. I'm not sure multiple 245$a's even technically _is_ "bad data". It's not neccesarily AACR2, but I found out the hard way recently on another project that we have LOTS of non-AACR2 data in our catalog, including AACR1 data, pre-AACR data, and data cataloged to rare books and manuscripts standards, none of which our catalogers actually consider 'bad'. So Ross suggests one answer, changing it to: title_t = custom, removeTrailingPunct(245a), first So it'll ignore the second 245. Another option might be figuring out how to set the solrmarc setup to concatenate two 245$a's, rather than ignore the second one, which would seem to me to be the actually appropriate thing to do in this case, barring letting title_t take multiple values. Is it possible to do that somehow? Anyone have an opinion on if either of these two things should be done in the standard out-of-the-box demo setup, to accomodate this kind of data? Jonathan Naomi Dushay wrote: > Jonathan, > > We have a TON of bad data. Lots of records with multiple 245a; lots > of records with other similar problems. > > We use solrmarc, and it nicely steps around these problems. There are > solrmarc lists; > > solrmarc-general at googlegroups.com > solrmarc-technical at googlegroups.com > > - Naomi > > > On Jun 22, 2009, at 1:44 PM, Jonathan Rochkind wrote: > > >> Trying to take one baby step at a time in getting a little demo of >> Blacklight with our demo, I'm trying to index a sample of our own >> local MARC records. >> >> I get an error from "rake app:index:marc", not sure exactly why, >> error below. Maybe errors in my MARC? There definitely _will_ be >> illegalities in my marc, would rather the indexer recovered from >> them somehow (or just skipped that record, if nothing else is >> possible), rather than punted the entire input. >> >> Should I switch to SolrMARC instead, is it a bit more forgiving? At >> one point I know I saw a pointer in the documentation to SolrMarc >> docs, to help you get started with solrmarc instead of the bundled >> ruby indexer... but now I can't seem to find it. >> >> Here's what "raek app:index:marc" is telling me (giving me an html >> error message even though it's a command-line rake task, which is a >> bit odd). >> rake aborted! >> >> >> >> Error 400 >> >>

HTTP ERROR: 400

ERROR: [53567] multiple values
>> encountered for non multiValued field title_t: [Honan's Handbook to
>> medical Europe, A ready reference book to the universities,
>> hospitals, clinics, laboratories and general medical work of the
>> principal cities of Europe]
>>

RequestURI=/solr/update

Powered by Jetty://


>> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From rh9ec at virginia.edu Mon Jun 22 17:40:09 2009 From: rh9ec at virginia.edu (Robert Haschart) Date: Mon, 22 Jun 2009 17:40:09 -0400 Subject: [Blacklight-development] error in marc indexing? In-Reply-To: <4A3FF59E.4030803@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> Message-ID: <4A3FFA39.3010403@virginia.edu> Jonathan, Rather than changing the schema to not specify multiValued, you also have a number of other options: change the spec for getting the title value from: title_t = 245a to : title_t = 245a, first which will get only the first occurance of a given field/subfield. or to: title_t = 245aa which will concatenate all 'a' subfields for a given 245 field and return them as a single entry. or even: title_t = 245aa, first which will handle instances with multiple 'a' subfields in a single 245 field as above, but still not die if multiple 245 fields are present. Lastly two features that were added to Solrmarc very recently (as in the demo code does not yet have it) could help your situation. One is that errors such as missing required field, duplicate, non-multiValued fields, or field with unknown (to solr) names, will generate an error message for that record, but allow the indexing to continue. Second there is a new standard "custom" index function called getSingleIndexEntry which could be used like this: title_t = custom, getSingleIndexEntry(245aa, true) It would process the 245aa field spec as above, and then if multiple entries results it would select the longest result to use for the title_t index entry, and if the second parameter to the function is true (and if marc.include_errors is enabled) it will generate a marc_error index entry containing the additional errorneous 245 field data. -Bob Haschart Jonathan Rochkind wrote: > Cool. I'm not sure multiple 245$a's even technically _is_ "bad data". > It's not neccesarily AACR2, but I found out the hard way recently on > another project that we have LOTS of non-AACR2 data in our catalog, > including AACR1 data, pre-AACR data, and data cataloged to rare books > and manuscripts standards, none of which our catalogers actually > consider 'bad'. > So Ross suggests one answer, changing it to: > > title_t = custom, removeTrailingPunct(245a), first > > So it'll ignore the second 245. Another option might be figuring out > how to set the solrmarc setup to concatenate two 245$a's, rather than > ignore the second one, which would seem to me to be the actually > appropriate thing to do in this case, barring letting title_t take > multiple values. Is it possible to do that somehow? > > Anyone have an opinion on if either of these two things should be done > in the standard out-of-the-box demo setup, to accomodate this kind of > data? > > Jonathan > > > > Naomi Dushay wrote: > >> Jonathan, >> >> We have a TON of bad data. Lots of records with multiple 245a; lots >> of records with other similar problems. >> >> We use solrmarc, and it nicely steps around these problems. There are >> solrmarc lists; >> >> solrmarc-general at googlegroups.com >> solrmarc-technical at googlegroups.com >> >> - Naomi >> >> >> On Jun 22, 2009, at 1:44 PM, Jonathan Rochkind wrote: >> >> >> >>> Trying to take one baby step at a time in getting a little demo of >>> Blacklight with our demo, I'm trying to index a sample of our own >>> local MARC records. >>> >>> I get an error from "rake app:index:marc", not sure exactly why, >>> error below. Maybe errors in my MARC? There definitely _will_ be >>> illegalities in my marc, would rather the indexer recovered from >>> them somehow (or just skipped that record, if nothing else is >>> possible), rather than punted the entire input. >>> >>> Should I switch to SolrMARC instead, is it a bit more forgiving? At >>> one point I know I saw a pointer in the documentation to SolrMarc >>> docs, to help you get started with solrmarc instead of the bundled >>> ruby indexer... but now I can't seem to find it. >>> >>> Here's what "raek app:index:marc" is telling me (giving me an html >>> error message even though it's a command-line rake task, which is a >>> bit odd). >>> rake aborted! >>> >>> >>> >>> Error 400 >>> >>>

HTTP ERROR: 400

ERROR: [53567] multiple values
>>> encountered for non multiValued field title_t: [Honan's Handbook to
>>> medical Europe, A ready reference book to the universities,
>>> hospitals, clinics, laboratories and general medical work of the
>>> principal cities of Europe]
>>>

RequestURI=/solr/update

>> href="http://jetty.mortbay.org/ >>> ">Powered by Jetty://


>>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From ndushay at stanford.edu Mon Jun 22 18:20:17 2009 From: ndushay at stanford.edu (Naomi Dushay) Date: Mon, 22 Jun 2009 15:20:17 -0700 Subject: [Blacklight-development] error in marc indexing? In-Reply-To: <4A3FF59E.4030803@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> Message-ID: <79F5FFC5-F42D-455A-BAA7-3F82A09E79D1@stanford.edu> Jonathan, Have you gotten blacklight up with the demo data in the demo index? That should be step 1. Step 2A: use solrmarc to generate a simple solr index from your institution's data (or a subset). This is a separate step. MARC data is complicated and varies widely from institution to institution. more below. Step 2B: hook blacklight up to the simple solr index from your institutions's data. Step 3: start tweaking what's in your index, and localizing blacklight for your data and for your institution. - Naomi On Jun 22, 2009, at 2:20 PM, Jonathan Rochkind wrote: > Cool. I'm not sure multiple 245$a's even technically _is_ "bad > data". It's not neccesarily AACR2, but I found out the hard way > recently on another project that we have LOTS of non-AACR2 data in > our catalog, including AACR1 data, pre-AACR data, and data cataloged > to rare books and manuscripts standards, none of which our > catalogers actually consider 'bad'. > So Ross suggests one answer, changing it to: > > title_t = custom, removeTrailingPunct(245a), first > > So it'll ignore the second 245. Another option might be figuring > out how to set the solrmarc setup to concatenate two 245$a's, rather > than ignore the second one, which would seem to me to be the > actually appropriate thing to do in this case, barring letting > title_t take multiple values. Is it possible to do that somehow? title_t = custom, removeTrailingPunct(245a), join or something close to that - see solrmarc docs distributed with solrmarc source code. > Anyone have an opinion on if either of these two things should be > done in the standard out-of-the-box demo setup, to accomodate this > kind of data? Answer: I don't think so. *Our* multiple 245s are generally 2 languages; the "other" one should be in an 880, per our cataloging guru. I think the solrmarc project is the context to discuss whether this is a common enough case to warrant further examples / documentation. Perhaps post a note to that list? There are about 4 examples of different solrmarc configurations that come with the solrmarc project code; our current blacklight oriented code is there, as is our deprecated vufind oriented indexing code. We don't allow multiple instances of solr index fields based on marc 245 fields. IMHO: the solrmarc project is about getting MARC data into a SOLR index. the Blacklight project is about discovery and display of data in a SOLR index (so far, in a library context ... but not nec. MARC data only) I'm as much at fault here as anyone, because it's easy to be lazy about using the proper list for the topic at hand. Just my $.02 - Naomi > > Jonathan > > > > Naomi Dushay wrote: >> Jonathan, >> >> We have a TON of bad data. Lots of records with multiple 245a; lots >> of records with other similar problems. >> >> We use solrmarc, and it nicely steps around these problems. There >> are >> solrmarc lists; >> >> solrmarc-general at googlegroups.com >> solrmarc-technical at googlegroups.com >> >> - Naomi >> >> >> On Jun 22, 2009, at 1:44 PM, Jonathan Rochkind wrote: >> >> >>> Trying to take one baby step at a time in getting a little demo of >>> Blacklight with our demo, I'm trying to index a sample of our own >>> local MARC records. >>> >>> I get an error from "rake app:index:marc", not sure exactly why, >>> error below. Maybe errors in my MARC? There definitely _will_ be >>> illegalities in my marc, would rather the indexer recovered from >>> them somehow (or just skipped that record, if nothing else is >>> possible), rather than punted the entire input. >>> >>> Should I switch to SolrMARC instead, is it a bit more forgiving? At >>> one point I know I saw a pointer in the documentation to SolrMarc >>> docs, to help you get started with solrmarc instead of the bundled >>> ruby indexer... but now I can't seem to find it. >>> >>> Here's what "raek app:index:marc" is telling me (giving me an html >>> error message even though it's a command-line rake task, which is a >>> bit odd). >>> rake aborted! >>> >>> >>> >>> Error 400 >>> >>>

HTTP ERROR: 400

ERROR: [53567] multiple values
>>> encountered for non multiValued field title_t: [Honan's Handbook to
>>> medical Europe, A ready reference book to the universities,
>>> hospitals, clinics, laboratories and general medical work of the
>>> principal cities of Europe]
>>>

RequestURI=/solr/update

Powered by Jetty://


>>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Mon Jun 22 18:20:40 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Mon, 22 Jun 2009 18:20:40 -0400 Subject: [Blacklight-development] error in marc indexing? In-Reply-To: <4A3FFA39.3010403@virginia.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> Message-ID: <4A4003B8.6030102@jhu.edu> Sweet, thanks a lot Robert, that's just what I needed to know. I believe that the 'right' thing to do, then, is: title_t = 245aa, first In the cases you were previously expecting, where there is only one 245$a, it will be indexed as before. In cases where there are multiple 245's or multiple $a's, then all of the "a"'s from the first 245 will be concatenated, and subsequent 245's will be ignored. I believe this is a better way to recover from that unexpected data than crashing entirely. If I submit a patch to the 'out of the box' Blacklight SOLRmarc config to change it like so, would you guys agree? Not sure if that patch goes to Blacklight project or SolrMARC project? Except now I'm confused as to what solrmarc.properties files is actually included with the Blacklight demo. The "vanilla blacklight demo" properties file in SolrMARC actually has: title_t = custom, removeTrailingPunct(245a) Is it possible to have title_t = custom, removeTrailingPunct(245aa), first ?? Should the properties file you get from a demo checkout be the SolrMarc "vanilla blacklight" example? I'm not sure what it is now...? Also, I could try to enhance the documentation on SolrMarc config to explain what "first" does (currently in there by example, but not actually described; are any other parameters valid there other than 'first'?), and to explain that you can double-up a subfield to mean "concatenate all such subfields from a given field" -- but I'm not sure I understand it well enough to write those docs? But maybe I'll just go go add that as I do understand it to the wiki? http://code.google.com/p/solrmarc/wiki/ConfiguringSolrMarc I guess I'd need to get added to the SolrMarc project to have permission to edit that wiki though. If anyone would like me to help enhance those docs, I'm _happy_ to do so, if someone can get me access to do so. Jonathan Robert Haschart wrote: > Jonathan, > > Rather than changing the schema to not specify multiValued, you also > have a number of other options: > > change the spec for getting the title value from: > > title_t = 245a > > to : > > title_t = 245a, first > > which will get only the first occurance of a given field/subfield. > > or to: > > title_t = 245aa > > which will concatenate all 'a' subfields for a given 245 field and > return them as a single entry. > > or even: > > title_t = 245aa, first > > which will handle instances with multiple 'a' subfields in a single 245 > field as above, but still not die if multiple 245 fields are present. > > Lastly two features that were added to Solrmarc very recently (as in the > demo code does not yet have it) could help your situation. One is that > errors such as missing required field, duplicate, non-multiValued > fields, or field with unknown (to solr) names, will generate an error > message for that record, but allow the indexing to continue. > Second there is a new standard "custom" index function called > getSingleIndexEntry which could be used like this: > > title_t = custom, getSingleIndexEntry(245aa, true) > > It would process the 245aa field spec as above, and then if multiple > entries results it would select the longest result to use for the > title_t index entry, and if the second parameter to the function is > true (and if marc.include_errors is enabled) it will generate a > marc_error index entry containing the additional errorneous 245 field > data. > > -Bob Haschart > > > Jonathan Rochkind wrote: > > >> Cool. I'm not sure multiple 245$a's even technically _is_ "bad data". >> It's not neccesarily AACR2, but I found out the hard way recently on >> another project that we have LOTS of non-AACR2 data in our catalog, >> including AACR1 data, pre-AACR data, and data cataloged to rare books >> and manuscripts standards, none of which our catalogers actually >> consider 'bad'. >> So Ross suggests one answer, changing it to: >> >> title_t = custom, removeTrailingPunct(245a), first >> >> So it'll ignore the second 245. Another option might be figuring out >> how to set the solrmarc setup to concatenate two 245$a's, rather than >> ignore the second one, which would seem to me to be the actually >> appropriate thing to do in this case, barring letting title_t take >> multiple values. Is it possible to do that somehow? >> >> Anyone have an opinion on if either of these two things should be done >> in the standard out-of-the-box demo setup, to accomodate this kind of >> data? >> >> Jonathan >> >> >> >> Naomi Dushay wrote: >> >> >>> Jonathan, >>> >>> We have a TON of bad data. Lots of records with multiple 245a; lots >>> of records with other similar problems. >>> >>> We use solrmarc, and it nicely steps around these problems. There are >>> solrmarc lists; >>> >>> solrmarc-general at googlegroups.com >>> solrmarc-technical at googlegroups.com >>> >>> - Naomi >>> >>> >>> On Jun 22, 2009, at 1:44 PM, Jonathan Rochkind wrote: >>> >>> >>> >>> >>>> Trying to take one baby step at a time in getting a little demo of >>>> Blacklight with our demo, I'm trying to index a sample of our own >>>> local MARC records. >>>> >>>> I get an error from "rake app:index:marc", not sure exactly why, >>>> error below. Maybe errors in my MARC? There definitely _will_ be >>>> illegalities in my marc, would rather the indexer recovered from >>>> them somehow (or just skipped that record, if nothing else is >>>> possible), rather than punted the entire input. >>>> >>>> Should I switch to SolrMARC instead, is it a bit more forgiving? At >>>> one point I know I saw a pointer in the documentation to SolrMarc >>>> docs, to help you get started with solrmarc instead of the bundled >>>> ruby indexer... but now I can't seem to find it. >>>> >>>> Here's what "raek app:index:marc" is telling me (giving me an html >>>> error message even though it's a command-line rake task, which is a >>>> bit odd). >>>> rake aborted! >>>> >>>> >>>> >>>> Error 400 >>>> >>>>

HTTP ERROR: 400

ERROR: [53567] multiple values
>>>> encountered for non multiValued field title_t: [Honan's Handbook to
>>>> medical Europe, A ready reference book to the universities,
>>>> hospitals, clinics, laboratories and general medical work of the
>>>> principal cities of Europe]
>>>>

RequestURI=/solr/update

>>> href="http://jetty.mortbay.org/ >>>> ">Powered by Jetty://


>>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From rochkind at jhu.edu Tue Jun 23 10:11:56 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 23 Jun 2009 10:11:56 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <4A4003B8.6030102@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> Message-ID: <4A40E2AC.3040604@jhu.edu> So, now I'm a bit confused about where I change the indexing configuration for SolrMarc in the demo app. The likely file I find that I think is controlling solrmarc when I run a "rake app:index:marc" is at: bl-demo/rails/vendor/plugins/blacklight/config/demo_index.properties But it doesn't seem right to me to have to change a file in the plugin source itself -- I want to leave the plugin pristine, so it can be easily updated when new versions come out, and make changes somewhere in my local app, right? What's the "right" way to make changes to the SolrMarc configuration, such that when I run "rake app:index:marc" in the demo app, they'll be used? Thanks for any help. Still curious if anyone has feedback on the below post about whether the 'standard' demo app solrmarc config should be changed? Jonathan Jonathan Rochkind wrote: > Sweet, thanks a lot Robert, that's just what I needed to know. > > I believe that the 'right' thing to do, then, is: > > title_t = 245aa, first > > > In the cases you were previously expecting, where there is only one > 245$a, it will be indexed as before. In cases where there are multiple > 245's or multiple $a's, then all of the "a"'s from the first 245 will be > concatenated, and subsequent 245's will be ignored. I believe this is a > better way to recover from that unexpected data than crashing entirely. > > If I submit a patch to the 'out of the box' Blacklight SOLRmarc config > to change it like so, would you guys agree? Not sure if that patch goes > to Blacklight project or SolrMARC project? > > Except now I'm confused as to what solrmarc.properties files is actually > included with the Blacklight demo. The "vanilla blacklight demo" > properties file in SolrMARC actually has: > > title_t = custom, removeTrailingPunct(245a) > > Is it possible to have > > title_t = custom, removeTrailingPunct(245aa), first ?? > > Should the properties file you get from a demo checkout be the SolrMarc > "vanilla blacklight" example? I'm not sure what it is now...? > > Also, I could try to enhance the documentation on SolrMarc config to > explain what "first" does (currently in there by example, but not > actually described; are any other parameters valid there other than > 'first'?), and to explain that you can double-up a subfield to mean > "concatenate all such subfields from a given field" -- but I'm not sure > I understand it well enough to write those docs? But maybe I'll just go > go add that as I do understand it to the wiki? > http://code.google.com/p/solrmarc/wiki/ConfiguringSolrMarc I guess I'd > need to get added to the SolrMarc project to have permission to edit > that wiki though. If anyone would like me to help enhance those docs, > I'm _happy_ to do so, if someone can get me access to do so. > > Jonathan > > Robert Haschart wrote: > >> Jonathan, >> >> Rather than changing the schema to not specify multiValued, you also >> have a number of other options: >> >> change the spec for getting the title value from: >> >> title_t = 245a >> >> to : >> >> title_t = 245a, first >> >> which will get only the first occurance of a given field/subfield. >> >> or to: >> >> title_t = 245aa >> >> which will concatenate all 'a' subfields for a given 245 field and >> return them as a single entry. >> >> or even: >> >> title_t = 245aa, first >> >> which will handle instances with multiple 'a' subfields in a single 245 >> field as above, but still not die if multiple 245 fields are present. >> >> Lastly two features that were added to Solrmarc very recently (as in the >> demo code does not yet have it) could help your situation. One is that >> errors such as missing required field, duplicate, non-multiValued >> fields, or field with unknown (to solr) names, will generate an error >> message for that record, but allow the indexing to continue. >> Second there is a new standard "custom" index function called >> getSingleIndexEntry which could be used like this: >> >> title_t = custom, getSingleIndexEntry(245aa, true) >> >> It would process the 245aa field spec as above, and then if multiple >> entries results it would select the longest result to use for the >> title_t index entry, and if the second parameter to the function is >> true (and if marc.include_errors is enabled) it will generate a >> marc_error index entry containing the additional errorneous 245 field >> data. >> >> -Bob Haschart >> >> >> Jonathan Rochkind wrote: >> >> >> >>> Cool. I'm not sure multiple 245$a's even technically _is_ "bad data". >>> It's not neccesarily AACR2, but I found out the hard way recently on >>> another project that we have LOTS of non-AACR2 data in our catalog, >>> including AACR1 data, pre-AACR data, and data cataloged to rare books >>> and manuscripts standards, none of which our catalogers actually >>> consider 'bad'. >>> So Ross suggests one answer, changing it to: >>> >>> title_t = custom, removeTrailingPunct(245a), first >>> >>> So it'll ignore the second 245. Another option might be figuring out >>> how to set the solrmarc setup to concatenate two 245$a's, rather than >>> ignore the second one, which would seem to me to be the actually >>> appropriate thing to do in this case, barring letting title_t take >>> multiple values. Is it possible to do that somehow? >>> >>> Anyone have an opinion on if either of these two things should be done >>> in the standard out-of-the-box demo setup, to accomodate this kind of >>> data? >>> >>> Jonathan >>> >>> >>> >>> Naomi Dushay wrote: >>> >>> >>> >>>> Jonathan, >>>> >>>> We have a TON of bad data. Lots of records with multiple 245a; lots >>>> of records with other similar problems. >>>> >>>> We use solrmarc, and it nicely steps around these problems. There are >>>> solrmarc lists; >>>> >>>> solrmarc-general at googlegroups.com >>>> solrmarc-technical at googlegroups.com >>>> >>>> - Naomi >>>> >>>> >>>> On Jun 22, 2009, at 1:44 PM, Jonathan Rochkind wrote: >>>> >>>> >>>> >>>> >>>> >>>>> Trying to take one baby step at a time in getting a little demo of >>>>> Blacklight with our demo, I'm trying to index a sample of our own >>>>> local MARC records. >>>>> >>>>> I get an error from "rake app:index:marc", not sure exactly why, >>>>> error below. Maybe errors in my MARC? There definitely _will_ be >>>>> illegalities in my marc, would rather the indexer recovered from >>>>> them somehow (or just skipped that record, if nothing else is >>>>> possible), rather than punted the entire input. >>>>> >>>>> Should I switch to SolrMARC instead, is it a bit more forgiving? At >>>>> one point I know I saw a pointer in the documentation to SolrMarc >>>>> docs, to help you get started with solrmarc instead of the bundled >>>>> ruby indexer... but now I can't seem to find it. >>>>> >>>>> Here's what "raek app:index:marc" is telling me (giving me an html >>>>> error message even though it's a command-line rake task, which is a >>>>> bit odd). >>>>> rake aborted! >>>>> >>>>> >>>>> >>>>> Error 400 >>>>> >>>>>

HTTP ERROR: 400

ERROR: [53567] multiple values
>>>>> encountered for non multiValued field title_t: [Honan's Handbook to
>>>>> medical Europe, A ready reference book to the universities,
>>>>> hospitals, clinics, laboratories and general medical work of the
>>>>> principal cities of Europe]
>>>>>

RequestURI=/solr/update

>>>> href="http://jetty.mortbay.org/ >>>>> ">Powered by Jetty://


>>>>> _______________________________________________ >>>>> Blacklight-development mailing list >>>>> Blacklight-development at rubyforge.org >>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From rochkind at jhu.edu Tue Jun 23 10:33:09 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 23 Jun 2009 10:33:09 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <23b83f160906230720h4485aac2r2fe24acdf05a7b73@mail.gmail.com> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <23b83f160906230720h4485aac2r2fe24acdf05a7b73@mail.gmail.com> Message-ID: <4A40E7A5.6070800@jhu.edu> Right, of course I'll make local modifcations, but I thought the point of the plug-in was that my local modifications would be in the app, NOT in the plugin. But the demo_index.properties file is in the plugin. Modifying the rake task to take the properties file from somewhere else seems like a reasonable approach. Would like to get feedback from main BL developers to see if this is what is reccommended, or what. Ross Singer wrote: > Jonathan, there is going to have to be some local modification somewhere... the notion of a 'pristine' Blacklight doesn't make sense, since 'out of the box' Blacklight isn't going to have any notion of your local context, data or needs. > > Either you'll have to change the demo_index.properties file or the solr_marc.rake file. The former seems simpler and more convenient to me. Alternately, I suppose you could clone the solr_marc.rake, put it in your main Rails app and modify it to use some copy of your *.properties. > > After all, all of this is solely for the BL /demo/. Stanford and, I assume, UVA probably have fairly different indexing routines than this for this actual instances. They are also probably fairly different from each other. > > -Ross. > > On Tue, Jun 23, 2009 at 10:11 AM, Jonathan Rochkind > wrote: > So, now I'm a bit confused about where I change the indexing configuration for SolrMarc in the demo app. > > The likely file I find that I think is controlling solrmarc when I run a "rake app:index:marc" is at: > > bl-demo/rails/vendor/plugins/blacklight/config/demo_index.properties > > But it doesn't seem right to me to have to change a file in the plugin source itself -- I want to leave the plugin pristine, so it can be easily updated when new versions come out, and make changes somewhere in my local app, right? > > What's the "right" way to make changes to the SolrMarc configuration, such that when I run "rake app:index:marc" in the demo app, they'll be used? > > Thanks for any help. Still curious if anyone has feedback on the below post about whether the 'standard' demo app solrmarc config should be changed? > > Jonathan > > > Jonathan Rochkind wrote: > Sweet, thanks a lot Robert, that's just what I needed to know. > > I believe that the 'right' thing to do, then, is: > > title_t = 245aa, first > > > In the cases you were previously expecting, where there is only one > 245$a, it will be indexed as before. In cases where there are multiple > 245's or multiple $a's, then all of the "a"'s from the first 245 will be > concatenated, and subsequent 245's will be ignored. I believe this is a > better way to recover from that unexpected data than crashing entirely. > > If I submit a patch to the 'out of the box' Blacklight SOLRmarc config > to change it like so, would you guys agree? Not sure if that patch goes > to Blacklight project or SolrMARC project? > > Except now I'm confused as to what solrmarc.properties files is actually > included with the Blacklight demo. The "vanilla blacklight demo" > properties file in SolrMARC actually has: > > title_t = custom, removeTrailingPunct(245a) > > Is it possible to have > > title_t = custom, removeTrailingPunct(245aa), first ?? > > Should the properties file you get from a demo checkout be the SolrMarc > "vanilla blacklight" example? I'm not sure what it is now...? > > Also, I could try to enhance the documentation on SolrMarc config to > explain what "first" does (currently in there by example, but not > actually described; are any other parameters valid there other than > 'first'?), and to explain that you can double-up a subfield to mean > "concatenate all such subfields from a given field" -- but I'm not sure > I understand it well enough to write those docs? But maybe I'll just go > go add that as I do understand it to the wiki? > http://code.google.com/p/solrmarc/wiki/ConfiguringSolrMarc I guess I'd > need to get added to the SolrMarc project to have permission to edit > that wiki though. If anyone would like me to help enhance those docs, > I'm _happy_ to do so, if someone can get me access to do so. > > Jonathan > > Robert Haschart wrote: > > Jonathan, > > Rather than changing the schema to not specify multiValued, you also > have a number of other options: > > change the spec for getting the title value from: > > title_t = 245a > > to : > > title_t = 245a, first > > which will get only the first occurance of a given field/subfield. > > or to: > > title_t = 245aa > > which will concatenate all 'a' subfields for a given 245 field and > return them as a single entry. > > or even: > > title_t = 245aa, first > > which will handle instances with multiple 'a' subfields in a single 245 > field as above, but still not die if multiple 245 fields are present. > > Lastly two features that were added to Solrmarc very recently (as in the > demo code does not yet have it) could help your situation. One is that > errors such as missing required field, duplicate, non-multiValued > fields, or field with unknown (to solr) names, will generate an error > message for that record, but allow the indexing to continue. > Second there is a new standard "custom" index function called > getSingleIndexEntry which could be used like this: > > title_t = custom, getSingleIndexEntry(245aa, true) > > It would process the 245aa field spec as above, and then if multiple > entries results it would select the longest result to use for the > title_t index entry, and if the second parameter to the function is > true (and if marc.include_errors is enabled) it will generate a > marc_error index entry containing the additional errorneous 245 field > data. > > -Bob Haschart > > > Jonathan Rochkind wrote: > > > > Cool. I'm not sure multiple 245$a's even technically _is_ "bad data". > It's not neccesarily AACR2, but I found out the hard way recently on > another project that we have LOTS of non-AACR2 data in our catalog, > including AACR1 data, pre-AACR data, and data cataloged to rare books > and manuscripts standards, none of which our catalogers actually > consider 'bad'. > So Ross suggests one answer, changing it to: > > title_t = custom, removeTrailingPunct(245a), first > > So it'll ignore the second 245. Another option might be figuring out > how to set the solrmarc setup to concatenate two 245$a's, rather than > ignore the second one, which would seem to me to be the actually > appropriate thing to do in this case, barring letting title_t take > multiple values. Is it possible to do that somehow? > > Anyone have an opinion on if either of these two things should be done > in the standard out-of-the-box demo setup, to accomodate this kind of > data? > > Jonathan > > > > Naomi Dushay wrote: > > > > Jonathan, > > We have a TON of bad data. Lots of records with multiple 245a; lots > of records with other similar problems. > > We use solrmarc, and it nicely steps around these problems. There are > solrmarc lists; > > solrmarc-general at googlegroups.com > solrmarc-technical at googlegroups.com > > - Naomi > > > On Jun 22, 2009, at 1:44 PM, Jonathan Rochkind wrote: > > > > > > Trying to take one baby step at a time in getting a little demo of > Blacklight with our demo, I'm trying to index a sample of our own > local MARC records. > > I get an error from "rake app:index:marc", not sure exactly why, > error below. Maybe errors in my MARC? There definitely _will_ be > illegalities in my marc, would rather the indexer recovered from > them somehow (or just skipped that record, if nothing else is > possible), rather than punted the entire input. > > Should I switch to SolrMARC instead, is it a bit more forgiving? At > one point I know I saw a pointer in the documentation to SolrMarc > docs, to help you get started with solrmarc instead of the bundled > ruby indexer... but now I can't seem to find it. > > Here's what "raek app:index:marc" is telling me (giving me an html > error message even though it's a command-line rake task, which is a > bit odd). > rake aborted! > > > > Error 400 > >

HTTP ERROR: 400

ERROR: [53567] multiple values
> encountered for non multiValued field title_t: [Honan's Handbook to
> medical Europe, A ready reference book to the universities,
> hospitals, clinics, laboratories and general medical work of the
> principal cities of Europe]
>

RequestURI=/solr/update

href="http://jetty.mortbay.org/ > ">Powered by Jetty://


> _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > From rochkind at jhu.edu Tue Jun 23 10:44:05 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 23 Jun 2009 10:44:05 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <23b83f160906230743p4c6373e0x256c659c0bf5c925@mail.gmail.com> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <23b83f160906230720h4485aac2r2fe24acdf05a7b73@mail.gmail.com> <4A40E7A5.6070800@jhu.edu> <23b83f160906230743p4c6373e0x256c659c0bf5c925@mail.gmail.com> Message-ID: <4A40EA35.5010101@jhu.edu> If it's in the app, I don't know where it is. Like I said in my original email, the properties file I found is at: bl-demo/rails/vendor/plugins/blacklight/config/demo_index.properties That's in the plugin. If I'm confused, then I'd definitely appreciate someone un-confusing me, but so far I'm just getting more confused. Jonathan Ross Singer wrote: > It is? Mine is in the app, although maybe I modified that at some point for Jangle. > > The rake tasks make some assumptions about where to find files (that I think can be overridden -- if not, they should be), which might be a way to go. > > -Ross. > > On Tue, Jun 23, 2009 at 10:33 AM, Jonathan Rochkind > wrote: > Right, of course I'll make local modifcations, but I thought the point of the plug-in was that my local modifications would be in the app, NOT in the plugin. > > But the demo_index.properties file is in the plugin. > Modifying the rake task to take the properties file from somewhere else seems like a reasonable approach. Would like to get feedback from main BL developers to see if this is what is reccommended, or what. > Ross Singer wrote: > Jonathan, there is going to have to be some local modification somewhere... the notion of a 'pristine' Blacklight doesn't make sense, since 'out of the box' Blacklight isn't going to have any notion of your local context, data or needs. > > Either you'll have to change the demo_index.properties file or the solr_marc.rake file. The former seems simpler and more convenient to me. Alternately, I suppose you could clone the solr_marc.rake, put it in your main Rails app and modify it to use some copy of your *.properties. > > After all, all of this is solely for the BL /demo/. Stanford and, I assume, UVA probably have fairly different indexing routines than this for this actual instances. They are also probably fairly different from each other. > > -Ross. > > On Tue, Jun 23, 2009 at 10:11 AM, Jonathan Rochkind >> wrote: > So, now I'm a bit confused about where I change the indexing configuration for SolrMarc in the demo app. > > The likely file I find that I think is controlling solrmarc when I run a "rake app:index:marc" is at: > > bl-demo/rails/vendor/plugins/blacklight/config/demo_index.properties > > But it doesn't seem right to me to have to change a file in the plugin source itself -- I want to leave the plugin pristine, so it can be easily updated when new versions come out, and make changes somewhere in my local app, right? > > What's the "right" way to make changes to the SolrMarc configuration, such that when I run "rake app:index:marc" in the demo app, they'll be used? > > Thanks for any help. Still curious if anyone has feedback on the below post about whether the 'standard' demo app solrmarc config should be changed? > > Jonathan > > > Jonathan Rochkind wrote: > Sweet, thanks a lot Robert, that's just what I needed to know. > > I believe that the 'right' thing to do, then, is: > > title_t = 245aa, first > > > In the cases you were previously expecting, where there is only one > 245$a, it will be indexed as before. In cases where there are multiple > 245's or multiple $a's, then all of the "a"'s from the first 245 will be > concatenated, and subsequent 245's will be ignored. I believe this is a > better way to recover from that unexpected data than crashing entirely. > > If I submit a patch to the 'out of the box' Blacklight SOLRmarc config > to change it like so, would you guys agree? Not sure if that patch goes > to Blacklight project or SolrMARC project? > > Except now I'm confused as to what solrmarc.properties files is actually > included with the Blacklight demo. The "vanilla blacklight demo" > properties file in SolrMARC actually has: > > title_t = custom, removeTrailingPunct(245a) > > Is it possible to have > > title_t = custom, removeTrailingPunct(245aa), first ?? > > Should the properties file you get from a demo checkout be the SolrMarc > "vanilla blacklight" example? I'm not sure what it is now...? > > Also, I could try to enhance the documentation on SolrMarc config to > explain what "first" does (currently in there by example, but not > actually described; are any other parameters valid there other than > 'first'?), and to explain that you can double-up a subfield to mean > "concatenate all such subfields from a given field" -- but I'm not sure > I understand it well enough to write those docs? But maybe I'll just go > go add that as I do understand it to the wiki? > http://code.google.com/p/solrmarc/wiki/ConfiguringSolrMarc I guess I'd > need to get added to the SolrMarc project to have permission to edit > that wiki though. If anyone would like me to help enhance those docs, > I'm _happy_ to do so, if someone can get me access to do so. > > Jonathan > > Robert Haschart wrote: > > Jonathan, > > Rather than changing the schema to not specify multiValued, you also > have a number of other options: > > change the spec for getting the title value from: > > title_t = 245a > > to : > > title_t = 245a, first > > which will get only the first occurance of a given field/subfield. > > or to: > > title_t = 245aa > > which will concatenate all 'a' subfields for a given 245 field and > return them as a single entry. > > or even: > > title_t = 245aa, first > > which will handle instances with multiple 'a' subfields in a single 245 > field as above, but still not die if multiple 245 fields are present. > > Lastly two features that were added to Solrmarc very recently (as in the > demo code does not yet have it) could help your situation. One is that > errors such as missing required field, duplicate, non-multiValued > fields, or field with unknown (to solr) names, will generate an error > message for that record, but allow the indexing to continue. > Second there is a new standard "custom" index function called > getSingleIndexEntry which could be used like this: > > title_t = custom, getSingleIndexEntry(245aa, true) > > It would process the 245aa field spec as above, and then if multiple > entries results it would select the longest result to use for the > title_t index entry, and if the second parameter to the function is > true (and if marc.include_errors is enabled) it will generate a > marc_error index entry containing the additional errorneous 245 field > data. > > -Bob Haschart > > > Jonathan Rochkind wrote: > > > > Cool. I'm not sure multiple 245$a's even technically _is_ "bad data". > It's not neccesarily AACR2, but I found out the hard way recently on > another project that we have LOTS of non-AACR2 data in our catalog, > including AACR1 data, pre-AACR data, and data cataloged to rare books > and manuscripts standards, none of which our catalogers actually > consider 'bad'. > So Ross suggests one answer, changing it to: > > title_t = custom, removeTrailingPunct(245a), first > > So it'll ignore the second 245. Another option might be figuring out > how to set the solrmarc setup to concatenate two 245$a's, rather than > ignore the second one, which would seem to me to be the actually > appropriate thing to do in this case, barring letting title_t take > multiple values. Is it possible to do that somehow? > > Anyone have an opinion on if either of these two things should be done > in the standard out-of-the-box demo setup, to accomodate this kind of > data? > > Jonathan > > > > Naomi Dushay wrote: > > > > Jonathan, > > We have a TON of bad data. Lots of records with multiple 245a; lots > of records with other similar problems. > > We use solrmarc, and it nicely steps around these problems. There are > solrmarc lists; > > solrmarc-general at googlegroups.com> > solrmarc-technical at googlegroups.com> > > > - Naomi > > > On Jun 22, 2009, at 1:44 PM, Jonathan Rochkind wrote: > > > > > > Trying to take one baby step at a time in getting a little demo of > Blacklight with our demo, I'm trying to index a sample of our own > local MARC records. > > I get an error from "rake app:index:marc", not sure exactly why, > error below. Maybe errors in my MARC? There definitely _will_ be > illegalities in my marc, would rather the indexer recovered from > them somehow (or just skipped that record, if nothing else is > possible), rather than punted the entire input. > > Should I switch to SolrMARC instead, is it a bit more forgiving? At > one point I know I saw a pointer in the documentation to SolrMarc > docs, to help you get started with solrmarc instead of the bundled > ruby indexer... but now I can't seem to find it. > > Here's what "raek app:index:marc" is telling me (giving me an html > error message even though it's a command-line rake task, which is a > bit odd). > rake aborted! > > > > Error 400 > >

HTTP ERROR: 400

ERROR: [53567] multiple values
> encountered for non multiValued field title_t: [Honan's Handbook to
> medical Europe, A ready reference book to the universities,
> hospitals, clinics, laboratories and general medical work of the
> principal cities of Europe]
>

RequestURI=/solr/update

href="http://jetty.mortbay.org/ > ">Powered by Jetty://


> _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org> > > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org> > > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org> > > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org> > > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org> > > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org> > > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > From rochkind at jhu.edu Tue Jun 23 11:05:49 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 23 Jun 2009 11:05:49 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <9B55AB03-FE07-4BF4-B402-EDB22BF4C332@virginia.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <23b83f160906230720h4485aac2r2fe24acdf05a7b73@mail.gmail.com> <9B55AB03-FE07-4BF4-B402-EDB22BF4C332@virginia.edu> Message-ID: <4A40EF4D.9010201@jhu.edu> Yes, that does help, thanks, that makes sense. It looks like the 'stock' solr_marc.rake task lives in the plugin (/lib/tasks/solr_marc.rake). I guess UVa and Stanford aren't actually using that 'stock' rake task? Should I add instructions to a wiki page somewhere on this? As a more medium-term solution, if I modified the solr_marc.rake task that lives in the plugin to take some configuration options from the app instead, so the rake task itself doesn't need to be cloned (and if it's later updated in the distro one can easily get those updates) ... would the BL community be amenable to that patch? Jonathan Bess Sadler wrote: > I'd recommend you clone the solr_marc.rake task, put it in your rails app, call it JHU_solrmarc.rake or whatever, and commit it to your repository. Now you've got the kernel of the JHU Blacklight instance, with your local indexing customizations. The idea behind the demo app is to give you an easy way to get started. You'll want to customize the demo app for local look and feel and behaviors, and quickly it will stop being "the blacklight demo app" (which no one at JHU would find very useful) and start being "the JHU blacklight installation". The idea is to keep your local changes in the parent app, so that when you want to you can upgrade the plugin (script/plugin -v install http://blacklight.rubyforge.org/svn/trunk/rails/vendor/plugins/blacklight/ --force) and take advantage of any shared development without writing over your local customizations. Any changes you make within the vendor/plugins/blacklight directory will be lost (unless you wanted to do a painful merge. *ugh*) the next time you upgrade. That's what this system is designed to avoid. > > There's going to be lots of stuff in the plugin that you want to customize and change. The right way to do that is to copy the file in question into the JHU implementation (thus overriding the plugin behavior) and make your customizations there. In most cases the file will need to have the same name in the same parallel file structure in order to override cleanly, but in the case of rake tasks you may need to call it something new. I'm not sure how this kind of over-riding changes for rake tasks. > > If you want to see an example of a customized Blacklight instance you can look at NWDA: http://github.com/bess/northwest-digital-archives/tree/master > You can see we have a local app.rake task for indexing NWDA data: http://github.com/bess/northwest-digital-archives/blob/542fa27d0a5e0ccd96e6e52dd1e52a4f08803fc9/lib/tasks/app.rake > > It started off as a copy of the rake task from the plugin, but it's been substantially re-written since then. > > Does that help? > > Bess > > > > On Jun 23, 2009, at 10:20 AM, Ross Singer wrote: > > Jonathan, there is going to have to be some local modification somewhere... the notion of a 'pristine' Blacklight doesn't make sense, since 'out of the box' Blacklight isn't going to have any notion of your local context, data or needs. > > Either you'll have to change the demo_index.properties file or the solr_marc.rake file. The former seems simpler and more convenient to me. Alternately, I suppose you could clone the solr_marc.rake, put it in your main Rails app and modify it to use some copy of your *.properties. > > After all, all of this is solely for the BL /demo/. Stanford and, I assume, UVA probably have fairly different indexing routines than this for this actual instances. They are also probably fairly different from each other. > > -Ross. > > On Tue, Jun 23, 2009 at 10:11 AM, Jonathan Rochkind > wrote: > So, now I'm a bit confused about where I change the indexing configuration for SolrMarc in the demo app. > > The likely file I find that I think is controlling solrmarc when I run a "rake app:index:marc" is at: > > bl-demo/rails/vendor/plugins/blacklight/config/demo_index.properties > > But it doesn't seem right to me to have to change a file in the plugin source itself -- I want to leave the plugin pristine, so it can be easily updated when new versions come out, and make changes somewhere in my local app, right? > > What's the "right" way to make changes to the SolrMarc configuration, such that when I run "rake app:index:marc" in the demo app, they'll be used? > > Thanks for any help. Still curious if anyone has feedback on the below post about whether the 'standard' demo app solrmarc config should be changed? > > Jonathan > > > Jonathan Rochkind wrote: > Sweet, thanks a lot Robert, that's just what I needed to know. > > I believe that the 'right' thing to do, then, is: > > title_t = 245aa, first > > > In the cases you were previously expecting, where there is only one > 245$a, it will be indexed as before. In cases where there are multiple > 245's or multiple $a's, then all of the "a"'s from the first 245 will be > concatenated, and subsequent 245's will be ignored. I believe this is a > better way to recover from that unexpected data than crashing entirely. > > If I submit a patch to the 'out of the box' Blacklight SOLRmarc config > to change it like so, would you guys agree? Not sure if that patch goes > to Blacklight project or SolrMARC project? > > Except now I'm confused as to what solrmarc.properties files is actually > included with the Blacklight demo. The "vanilla blacklight demo" > properties file in SolrMARC actually has: > > title_t = custom, removeTrailingPunct(245a) > > Is it possible to have > > title_t = custom, removeTrailingPunct(245aa), first ?? > > Should the properties file you get from a demo checkout be the SolrMarc > "vanilla blacklight" example? I'm not sure what it is now...? > > Also, I could try to enhance the documentation on SolrMarc config to > explain what "first" does (currently in there by example, but not > actually described; are any other parameters valid there other than > 'first'?), and to explain that you can double-up a subfield to mean > "concatenate all such subfields from a given field" -- but I'm not sure > I understand it well enough to write those docs? But maybe I'll just go > go add that as I do understand it to the wiki? > http://code.google.com/p/solrmarc/wiki/ConfiguringSolrMarc I guess I'd > need to get added to the SolrMarc project to have permission to edit > that wiki though. If anyone would like me to help enhance those docs, > I'm _happy_ to do so, if someone can get me access to do so. > > Jonathan > > Robert Haschart wrote: > > Jonathan, > > Rather than changing the schema to not specify multiValued, you also > have a number of other options: > > change the spec for getting the title value from: > > title_t = 245a > > to : > > title_t = 245a, first > > which will get only the first occurance of a given field/subfield. > > or to: > > title_t = 245aa > > which will concatenate all 'a' subfields for a given 245 field and > return them as a single entry. > > or even: > > title_t = 245aa, first > > which will handle instances with multiple 'a' subfields in a single 245 > field as above, but still not die if multiple 245 fields are present. > > Lastly two features that were added to Solrmarc very recently (as in the > demo code does not yet have it) could help your situation. One is that > errors such as missing required field, duplicate, non-multiValued > fields, or field with unknown (to solr) names, will generate an error > message for that record, but allow the indexing to continue. > Second there is a new standard "custom" index function called > getSingleIndexEntry which could be used like this: > > title_t = custom, getSingleIndexEntry(245aa, true) > > It would process the 245aa field spec as above, and then if multiple > entries results it would select the longest result to use for the > title_t index entry, and if the second parameter to the function is > true (and if marc.include_errors is enabled) it will generate a > marc_error index entry containing the additional errorneous 245 field > data. > > -Bob Haschart > > > Jonathan Rochkind wrote: > > > > Cool. I'm not sure multiple 245$a's even technically _is_ "bad data". > It's not neccesarily AACR2, but I found out the hard way recently on > another project that we have LOTS of non-AACR2 data in our catalog, > including AACR1 data, pre-AACR data, and data cataloged to rare books > and manuscripts standards, none of which our catalogers actually > consider 'bad'. > So Ross suggests one answer, changing it to: > > title_t = custom, removeTrailingPunct(245a), first > > So it'll ignore the second 245. Another option might be figuring out > how to set the solrmarc setup to concatenate two 245$a's, rather than > ignore the second one, which would seem to me to be the actually > appropriate thing to do in this case, barring letting title_t take > multiple values. Is it possible to do that somehow? > > Anyone have an opinion on if either of these two things should be done > in the standard out-of-the-box demo setup, to accomodate this kind of > data? > > Jonathan > > > > Naomi Dushay wrote: > > > > Jonathan, > > We have a TON of bad data. Lots of records with multiple 245a; lots > of records with other similar problems. > > We use solrmarc, and it nicely steps around these problems. There are > solrmarc lists; > > solrmarc-general at googlegroups.com > solrmarc-technical at googlegroups.com > > - Naomi > > > On Jun 22, 2009, at 1:44 PM, Jonathan Rochkind wrote: > > > > > > Trying to take one baby step at a time in getting a little demo of > Blacklight with our demo, I'm trying to index a sample of our own > local MARC records. > > I get an error from "rake app:index:marc", not sure exactly why, > error below. Maybe errors in my MARC? There definitely _will_ be > illegalities in my marc, would rather the indexer recovered from > them somehow (or just skipped that record, if nothing else is > possible), rather than punted the entire input. > > Should I switch to SolrMARC instead, is it a bit more forgiving? At > one point I know I saw a pointer in the documentation to SolrMarc > docs, to help you get started with solrmarc instead of the bundled > ruby indexer... but now I can't seem to find it. > > Here's what "raek app:index:marc" is telling me (giving me an html > error message even though it's a command-line rake task, which is a > bit odd). > rake aborted! > > > > Error 400 > >

HTTP ERROR: 400

ERROR: [53567] multiple values
> encountered for non multiValued field title_t: [Honan's Handbook to
> medical Europe, A ready reference book to the universities,
> hospitals, clinics, laboratories and general medical work of the
> principal cities of Europe]
>

RequestURI=/solr/update

href="http://jetty.mortbay.org/ > ">Powered by Jetty://


> _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > > > From rochkind at jhu.edu Tue Jun 23 11:12:16 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 23 Jun 2009 11:12:16 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <4A40EF4D.9010201@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <23b83f160906230720h4485aac2r2fe24acdf05a7b73@mail.gmail.com> <9B55AB03-FE07-4BF4-B402-EDB22BF4C332@virginia.edu> <4A40EF4D.9010201@jhu.edu> Message-ID: <4A40F0D0.5080005@jhu.edu> Aha! It looks like no patch is needed, the solr_marc.rake task already is capable of looking .properties file from an ENV variable. Looking at the source. Very nice, whoever wrote that already considered this. So it shouldn't be neccesary to make a local copy of the rake task, rather one can just set the ENV variable to point to your own properties file. I'll investigate if it's possible to set an ENV variable from the local Rails app environment.rb file, which is what would be most convenient for me. Hopefully it is, as it makes sense to me that it's easiest to set this kind of 'override' in environment.rb, and not have to do deal with bash profiles and such. And once I figure out the most convenient way to do so, I'll add a wiki page? Sound good? Jonathan Jonathan Rochkind wrote: > Yes, that does help, thanks, that makes sense. It looks like the > 'stock' solr_marc.rake task lives in the plugin > (/lib/tasks/solr_marc.rake). I guess UVa and Stanford aren't actually > using that 'stock' rake task? > > Should I add instructions to a wiki page somewhere on this? > > As a more medium-term solution, if I modified the solr_marc.rake task > that lives in the plugin to take some configuration options from the app > instead, so the rake task itself doesn't need to be cloned (and if it's > later updated in the distro one can easily get those updates) ... would > the BL community be amenable to that patch? > > Jonathan > > Bess Sadler wrote: > >> I'd recommend you clone the solr_marc.rake task, put it in your rails app, call it JHU_solrmarc.rake or whatever, and commit it to your repository. Now you've got the kernel of the JHU Blacklight instance, with your local indexing customizations. The idea behind the demo app is to give you an easy way to get started. You'll want to customize the demo app for local look and feel and behaviors, and quickly it will stop being "the blacklight demo app" (which no one at JHU would find very useful) and start being "the JHU blacklight installation". The idea is to keep your local changes in the parent app, so that when you want to you can upgrade the plugin (script/plugin -v install http://blacklight.rubyforge.org/svn/trunk/rails/vendor/plugins/blacklight/ --force) and take advantage of any shared development without writing over your local customizations. Any changes you make within the vendor/plugins/blacklight directory will be lost (unless you wanted to do a painful merge. *ugh*) the next time you upgrade. That's what this system is designed to avoid. >> >> There's going to be lots of stuff in the plugin that you want to customize and change. The right way to do that is to copy the file in question into the JHU implementation (thus overriding the plugin behavior) and make your customizations there. In most cases the file will need to have the same name in the same parallel file structure in order to override cleanly, but in the case of rake tasks you may need to call it something new. I'm not sure how this kind of over-riding changes for rake tasks. >> >> If you want to see an example of a customized Blacklight instance you can look at NWDA: http://github.com/bess/northwest-digital-archives/tree/master >> You can see we have a local app.rake task for indexing NWDA data: http://github.com/bess/northwest-digital-archives/blob/542fa27d0a5e0ccd96e6e52dd1e52a4f08803fc9/lib/tasks/app.rake >> >> It started off as a copy of the rake task from the plugin, but it's been substantially re-written since then. >> >> Does that help? >> >> Bess >> >> >> >> On Jun 23, 2009, at 10:20 AM, Ross Singer wrote: >> >> Jonathan, there is going to have to be some local modification somewhere... the notion of a 'pristine' Blacklight doesn't make sense, since 'out of the box' Blacklight isn't going to have any notion of your local context, data or needs. >> >> Either you'll have to change the demo_index.properties file or the solr_marc.rake file. The former seems simpler and more convenient to me. Alternately, I suppose you could clone the solr_marc.rake, put it in your main Rails app and modify it to use some copy of your *.properties. >> >> After all, all of this is solely for the BL /demo/. Stanford and, I assume, UVA probably have fairly different indexing routines than this for this actual instances. They are also probably fairly different from each other. >> >> -Ross. >> >> On Tue, Jun 23, 2009 at 10:11 AM, Jonathan Rochkind > wrote: >> So, now I'm a bit confused about where I change the indexing configuration for SolrMarc in the demo app. >> >> The likely file I find that I think is controlling solrmarc when I run a "rake app:index:marc" is at: >> >> bl-demo/rails/vendor/plugins/blacklight/config/demo_index.properties >> >> But it doesn't seem right to me to have to change a file in the plugin source itself -- I want to leave the plugin pristine, so it can be easily updated when new versions come out, and make changes somewhere in my local app, right? >> >> What's the "right" way to make changes to the SolrMarc configuration, such that when I run "rake app:index:marc" in the demo app, they'll be used? >> >> Thanks for any help. Still curious if anyone has feedback on the below post about whether the 'standard' demo app solrmarc config should be changed? >> >> Jonathan >> >> >> Jonathan Rochkind wrote: >> Sweet, thanks a lot Robert, that's just what I needed to know. >> >> I believe that the 'right' thing to do, then, is: >> >> title_t = 245aa, first >> >> >> In the cases you were previously expecting, where there is only one >> 245$a, it will be indexed as before. In cases where there are multiple >> 245's or multiple $a's, then all of the "a"'s from the first 245 will be >> concatenated, and subsequent 245's will be ignored. I believe this is a >> better way to recover from that unexpected data than crashing entirely. >> >> If I submit a patch to the 'out of the box' Blacklight SOLRmarc config >> to change it like so, would you guys agree? Not sure if that patch goes >> to Blacklight project or SolrMARC project? >> >> Except now I'm confused as to what solrmarc.properties files is actually >> included with the Blacklight demo. The "vanilla blacklight demo" >> properties file in SolrMARC actually has: >> >> title_t = custom, removeTrailingPunct(245a) >> >> Is it possible to have >> >> title_t = custom, removeTrailingPunct(245aa), first ?? >> >> Should the properties file you get from a demo checkout be the SolrMarc >> "vanilla blacklight" example? I'm not sure what it is now...? >> >> Also, I could try to enhance the documentation on SolrMarc config to >> explain what "first" does (currently in there by example, but not >> actually described; are any other parameters valid there other than >> 'first'?), and to explain that you can double-up a subfield to mean >> "concatenate all such subfields from a given field" -- but I'm not sure >> I understand it well enough to write those docs? But maybe I'll just go >> go add that as I do understand it to the wiki? >> http://code.google.com/p/solrmarc/wiki/ConfiguringSolrMarc I guess I'd >> need to get added to the SolrMarc project to have permission to edit >> that wiki though. If anyone would like me to help enhance those docs, >> I'm _happy_ to do so, if someone can get me access to do so. >> >> Jonathan >> >> Robert Haschart wrote: >> >> Jonathan, >> >> Rather than changing the schema to not specify multiValued, you also >> have a number of other options: >> >> change the spec for getting the title value from: >> >> title_t = 245a >> >> to : >> >> title_t = 245a, first >> >> which will get only the first occurance of a given field/subfield. >> >> or to: >> >> title_t = 245aa >> >> which will concatenate all 'a' subfields for a given 245 field and >> return them as a single entry. >> >> or even: >> >> title_t = 245aa, first >> >> which will handle instances with multiple 'a' subfields in a single 245 >> field as above, but still not die if multiple 245 fields are present. >> >> Lastly two features that were added to Solrmarc very recently (as in the >> demo code does not yet have it) could help your situation. One is that >> errors such as missing required field, duplicate, non-multiValued >> fields, or field with unknown (to solr) names, will generate an error >> message for that record, but allow the indexing to continue. >> Second there is a new standard "custom" index function called >> getSingleIndexEntry which could be used like this: >> >> title_t = custom, getSingleIndexEntry(245aa, true) >> >> It would process the 245aa field spec as above, and then if multiple >> entries results it would select the longest result to use for the >> title_t index entry, and if the second parameter to the function is >> true (and if marc.include_errors is enabled) it will generate a >> marc_error index entry containing the additional errorneous 245 field >> data. >> >> -Bob Haschart >> >> >> Jonathan Rochkind wrote: >> >> >> >> Cool. I'm not sure multiple 245$a's even technically _is_ "bad data". >> It's not neccesarily AACR2, but I found out the hard way recently on >> another project that we have LOTS of non-AACR2 data in our catalog, >> including AACR1 data, pre-AACR data, and data cataloged to rare books >> and manuscripts standards, none of which our catalogers actually >> consider 'bad'. >> So Ross suggests one answer, changing it to: >> >> title_t = custom, removeTrailingPunct(245a), first >> >> So it'll ignore the second 245. Another option might be figuring out >> how to set the solrmarc setup to concatenate two 245$a's, rather than >> ignore the second one, which would seem to me to be the actually >> appropriate thing to do in this case, barring letting title_t take >> multiple values. Is it possible to do that somehow? >> >> Anyone have an opinion on if either of these two things should be done >> in the standard out-of-the-box demo setup, to accomodate this kind of >> data? >> >> Jonathan >> >> >> >> Naomi Dushay wrote: >> >> >> >> Jonathan, >> >> We have a TON of bad data. Lots of records with multiple 245a; lots >> of records with other similar problems. >> >> We use solrmarc, and it nicely steps around these problems. There are >> solrmarc lists; >> >> solrmarc-general at googlegroups.com >> solrmarc-technical at googlegroups.com >> >> - Naomi >> >> >> On Jun 22, 2009, at 1:44 PM, Jonathan Rochkind wrote: >> >> >> >> >> >> Trying to take one baby step at a time in getting a little demo of >> Blacklight with our demo, I'm trying to index a sample of our own >> local MARC records. >> >> I get an error from "rake app:index:marc", not sure exactly why, >> error below. Maybe errors in my MARC? There definitely _will_ be >> illegalities in my marc, would rather the indexer recovered from >> them somehow (or just skipped that record, if nothing else is >> possible), rather than punted the entire input. >> >> Should I switch to SolrMARC instead, is it a bit more forgiving? At >> one point I know I saw a pointer in the documentation to SolrMarc >> docs, to help you get started with solrmarc instead of the bundled >> ruby indexer... but now I can't seem to find it. >> >> Here's what "raek app:index:marc" is telling me (giving me an html >> error message even though it's a command-line rake task, which is a >> bit odd). >> rake aborted! >> >> >> >> Error 400 >> >>

HTTP ERROR: 400

ERROR: [53567] multiple values
>> encountered for non multiValued field title_t: [Honan's Handbook to
>> medical Europe, A ready reference book to the universities,
>> hospitals, clinics, laboratories and general medical work of the
>> principal cities of Europe]
>>

RequestURI=/solr/update

> href="http://jetty.mortbay.org/ >> ">Powered by Jetty://


>> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> >> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> >> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> >> >> >> >> >> > > > From eos8d at virginia.edu Tue Jun 23 11:19:23 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Tue, 23 Jun 2009 11:19:23 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <4A40F0D0.5080005@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <23b83f160906230720h4485aac2r2fe24acdf05a7b73@mail.gmail.com> <9B55AB03-FE07-4BF4-B402-EDB22BF4C332@virginia.edu> <4A40EF4D.9010201@jhu.edu> <4A40F0D0.5080005@jhu.edu> Message-ID: <19BF5D95-D23C-441B-A04B-17AED2AAEDA2@virginia.edu> On 23-Jun-09, at 11:12 AM, Jonathan Rochkind wrote: > Aha! It looks like no patch is needed, the solr_marc.rake task already > is capable of looking .properties file from an ENV variable. > Looking at > the source. Very nice, whoever wrote that already considered this. > So it > shouldn't be neccesary to make a local copy of the rake task, rather > one > can just set the ENV variable to point to your own properties file. Nice. Jamie wrote that, I believe, and I haven't had a chance to do much except glance at it. UVA already had an indexing procedure in place so we're not using that rake task, and NWDA isn't indexing marc records so far, so not using it there either. Thanks for the offer to document this procedure, that would be great! Bess From jamie at dang.com Tue Jun 23 11:27:56 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Tue, 23 Jun 2009 11:27:56 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <4A40E2AC.3040604@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> Message-ID: rake solr:marc:index:info That will give you info you need to setup your config for the solrmarc indexer rake solr:marc:index is the actual command. There are two or three environment variables that the info task should indicate. We should get rid of rake app:index:ead and rake app:index:marc if they are not useful any longer. Jamie On Jun 23, 2009, at 10:11 AM, Jonathan Rochkind wrote: > So, now I'm a bit confused about where I change the indexing > configuration for SolrMarc in the demo app. > > The likely file I find that I think is controlling solrmarc when I > run a "rake app:index:marc" is at: > > bl-demo/rails/vendor/plugins/blacklight/config/demo_index.properties > > But it doesn't seem right to me to have to change a file in the > plugin source itself -- I want to leave the plugin pristine, so it > can be easily updated when new versions come out, and make changes > somewhere in my local app, right? > > What's the "right" way to make changes to the SolrMarc > configuration, such that when I run "rake app:index:marc" in the > demo app, they'll be used? > > Thanks for any help. Still curious if anyone has feedback on the > below post about whether the 'standard' demo app solrmarc config > should be changed? > > Jonathan > > > Jonathan Rochkind wrote: >> Sweet, thanks a lot Robert, that's just what I needed to know. >> >> I believe that the 'right' thing to do, then, is: >> >> title_t = 245aa, first >> >> >> In the cases you were previously expecting, where there is only one >> 245$a, it will be indexed as before. In cases where there are >> multiple >> 245's or multiple $a's, then all of the "a"'s from the first 245 >> will be >> concatenated, and subsequent 245's will be ignored. I believe this >> is a >> better way to recover from that unexpected data than crashing >> entirely. >> >> If I submit a patch to the 'out of the box' Blacklight SOLRmarc >> config >> to change it like so, would you guys agree? Not sure if that patch >> goes >> to Blacklight project or SolrMARC project? >> >> Except now I'm confused as to what solrmarc.properties files is >> actually >> included with the Blacklight demo. The "vanilla blacklight demo" >> properties file in SolrMARC actually has: >> >> title_t = custom, removeTrailingPunct(245a) >> >> Is it possible to have >> >> title_t = custom, removeTrailingPunct(245aa), first ?? >> >> Should the properties file you get from a demo checkout be the >> SolrMarc >> "vanilla blacklight" example? I'm not sure what it is now...? >> >> Also, I could try to enhance the documentation on SolrMarc config to >> explain what "first" does (currently in there by example, but not >> actually described; are any other parameters valid there other than >> 'first'?), and to explain that you can double-up a subfield to mean >> "concatenate all such subfields from a given field" -- but I'm not >> sure >> I understand it well enough to write those docs? But maybe I'll >> just go >> go add that as I do understand it to the wiki? >> http://code.google.com/p/solrmarc/wiki/ConfiguringSolrMarc I >> guess I'd >> need to get added to the SolrMarc project to have permission to edit >> that wiki though. If anyone would like me to help enhance those >> docs, >> I'm _happy_ to do so, if someone can get me access to do so. >> >> Jonathan >> >> Robert Haschart wrote: >> >>> Jonathan, >>> >>> Rather than changing the schema to not specify multiValued, you also >>> have a number of other options: >>> >>> change the spec for getting the title value from: >>> >>> title_t = 245a >>> >>> to : >>> >>> title_t = 245a, first >>> >>> which will get only the first occurance of a given field/subfield. >>> >>> or to: >>> >>> title_t = 245aa >>> >>> which will concatenate all 'a' subfields for a given 245 field and >>> return them as a single entry. >>> >>> or even: >>> >>> title_t = 245aa, first >>> >>> which will handle instances with multiple 'a' subfields in a >>> single 245 >>> field as above, but still not die if multiple 245 fields are >>> present. >>> >>> Lastly two features that were added to Solrmarc very recently (as >>> in the >>> demo code does not yet have it) could help your situation. One is >>> that >>> errors such as missing required field, duplicate, non-multiValued >>> fields, or field with unknown (to solr) names, will generate an >>> error >>> message for that record, but allow the indexing to continue. >>> Second there is a new standard "custom" index function called >>> getSingleIndexEntry which could be used like this: >>> >>> title_t = custom, getSingleIndexEntry(245aa, true) >>> >>> It would process the 245aa field spec as above, and then if multiple >>> entries results it would select the longest result to use for the >>> title_t index entry, and if the second parameter to the function >>> is >>> true (and if marc.include_errors is enabled) it will generate a >>> marc_error index entry containing the additional errorneous 245 >>> field >>> data. >>> >>> -Bob Haschart >>> >>> >>> Jonathan Rochkind wrote: >>> >>> >>> >>>> Cool. I'm not sure multiple 245$a's even technically _is_ "bad >>>> data". >>>> It's not neccesarily AACR2, but I found out the hard way recently >>>> on >>>> another project that we have LOTS of non-AACR2 data in our catalog, >>>> including AACR1 data, pre-AACR data, and data cataloged to rare >>>> books >>>> and manuscripts standards, none of which our catalogers actually >>>> consider 'bad'. >>>> So Ross suggests one answer, changing it to: >>>> >>>> title_t = custom, removeTrailingPunct(245a), first >>>> >>>> So it'll ignore the second 245. Another option might be figuring >>>> out >>>> how to set the solrmarc setup to concatenate two 245$a's, rather >>>> than >>>> ignore the second one, which would seem to me to be the actually >>>> appropriate thing to do in this case, barring letting title_t take >>>> multiple values. Is it possible to do that somehow? >>>> >>>> Anyone have an opinion on if either of these two things should be >>>> done >>>> in the standard out-of-the-box demo setup, to accomodate this >>>> kind of >>>> data? >>>> >>>> Jonathan >>>> >>>> >>>> >>>> Naomi Dushay wrote: >>>> >>>> >>>> >>>>> Jonathan, >>>>> >>>>> We have a TON of bad data. Lots of records with multiple 245a; >>>>> lots >>>>> of records with other similar problems. >>>>> >>>>> We use solrmarc, and it nicely steps around these problems. >>>>> There are >>>>> solrmarc lists; >>>>> >>>>> solrmarc-general at googlegroups.com >>>>> solrmarc-technical at googlegroups.com >>>>> >>>>> - Naomi >>>>> >>>>> >>>>> On Jun 22, 2009, at 1:44 PM, Jonathan Rochkind wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Trying to take one baby step at a time in getting a little demo >>>>>> of >>>>>> Blacklight with our demo, I'm trying to index a sample of our own >>>>>> local MARC records. >>>>>> >>>>>> I get an error from "rake app:index:marc", not sure exactly why, >>>>>> error below. Maybe errors in my MARC? There definitely _will_ be >>>>>> illegalities in my marc, would rather the indexer recovered from >>>>>> them somehow (or just skipped that record, if nothing else is >>>>>> possible), rather than punted the entire input. >>>>>> >>>>>> Should I switch to SolrMARC instead, is it a bit more >>>>>> forgiving? At >>>>>> one point I know I saw a pointer in the documentation to SolrMarc >>>>>> docs, to help you get started with solrmarc instead of the >>>>>> bundled >>>>>> ruby indexer... but now I can't seem to find it. >>>>>> >>>>>> Here's what "raek app:index:marc" is telling me (giving me an >>>>>> html >>>>>> error message even though it's a command-line rake task, which >>>>>> is a >>>>>> bit odd). >>>>>> rake aborted! >>>>>> >>>>>> >>>>>> >>>>>> Error 400 >>>>>> >>>>>>

HTTP ERROR: 400

ERROR: [53567] multiple values
>>>>>> encountered for non multiValued field title_t: [Honan's  
>>>>>> Handbook to
>>>>>> medical Europe, A ready reference book to the universities,
>>>>>> hospitals, clinics, laboratories and general medical work of the
>>>>>> principal cities of Europe]
>>>>>>

RequestURI=/solr/update

>>>>> href="http://jetty.mortbay.org/ >>>>>> ">Powered by Jetty://


>>>>>> _______________________________________________ >>>>>> Blacklight-development mailing list >>>>>> Blacklight-development at rubyforge.org >>>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Blacklight-development mailing list >>>>> Blacklight-development at rubyforge.org >>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Tue Jun 23 11:28:58 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 23 Jun 2009 11:28:58 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <19BF5D95-D23C-441B-A04B-17AED2AAEDA2@virginia.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <23b83f160906230720h4485aac2r2fe24acdf05a7b73@mail.gmail.com> <9B55AB03-FE07-4BF4-B402-EDB22BF4C332@virginia.edu> <4A40EF4D.9010201@jhu.edu> <4A40F0D0.5080005@jhu.edu> <19BF5D95-D23C-441B-A04B-17AED2AAEDA2@virginia.edu> Message-ID: <4A40F4BA.70301@jhu.edu> Actually, I think I do want to write a patch. The ENV variables the rake task is looking for are normally set on the command line. If no ENV variable for properties file is supplied, then the task looks in the plugin default location. I'd like to patch it so if it doesn't have an ENV variable set, it looks in a certain location in the _app_ first (config/solrmarc/*), and only if nothing is found there, uses the 'stock' one from the plugin itself. Make sense? I'll file the patch in JIRA. And document... I guess the way you have to do it now, and once/if the patch is accepted, change that. Bess Sadler wrote: > On 23-Jun-09, at 11:12 AM, Jonathan Rochkind wrote: > > >> Aha! It looks like no patch is needed, the solr_marc.rake task already >> is capable of looking .properties file from an ENV variable. >> Looking at >> the source. Very nice, whoever wrote that already considered this. >> So it >> shouldn't be neccesary to make a local copy of the rake task, rather >> one >> can just set the ENV variable to point to your own properties file. >> > > Nice. Jamie wrote that, I believe, and I haven't had a chance to do > much except glance at it. UVA already had an indexing procedure in > place so we're not using that rake task, and NWDA isn't indexing marc > records so far, so not using it there either. Thanks for the offer to > document this procedure, that would be great! > > Bess > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From rochkind at jhu.edu Tue Jun 23 11:30:16 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 23 Jun 2009 11:30:16 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> Message-ID: <4A40F508.30400@jhu.edu> Oh, wait a second. The index command I was using right now is 'rake app:index:marc'. I got that from the documentation. Should the documentation be changed to reccommend "rake solr:marc:index" instead? That documentation is actually in the source. I'll file a patch for that too; I already filed a couple documentation patches, not sure anyone is paying attention to them? Jamie Orchard-Hays wrote: > rake solr:marc:index:info > > That will give you info you need to setup your config for the solrmarc > indexer > > rake solr:marc:index > > is the actual command. > > There are two or three environment variables that the info task should > indicate. > > We should get rid of rake app:index:ead and rake app:index:marc if > they are not useful any longer. > > Jamie > On Jun 23, 2009, at 10:11 AM, Jonathan Rochkind wrote: > > >> So, now I'm a bit confused about where I change the indexing >> configuration for SolrMarc in the demo app. >> >> The likely file I find that I think is controlling solrmarc when I >> run a "rake app:index:marc" is at: >> >> bl-demo/rails/vendor/plugins/blacklight/config/demo_index.properties >> >> But it doesn't seem right to me to have to change a file in the >> plugin source itself -- I want to leave the plugin pristine, so it >> can be easily updated when new versions come out, and make changes >> somewhere in my local app, right? >> >> What's the "right" way to make changes to the SolrMarc >> configuration, such that when I run "rake app:index:marc" in the >> demo app, they'll be used? >> >> Thanks for any help. Still curious if anyone has feedback on the >> below post about whether the 'standard' demo app solrmarc config >> should be changed? >> >> Jonathan >> >> >> Jonathan Rochkind wrote: >> >>> Sweet, thanks a lot Robert, that's just what I needed to know. >>> >>> I believe that the 'right' thing to do, then, is: >>> >>> title_t = 245aa, first >>> >>> >>> In the cases you were previously expecting, where there is only one >>> 245$a, it will be indexed as before. In cases where there are >>> multiple >>> 245's or multiple $a's, then all of the "a"'s from the first 245 >>> will be >>> concatenated, and subsequent 245's will be ignored. I believe this >>> is a >>> better way to recover from that unexpected data than crashing >>> entirely. >>> >>> If I submit a patch to the 'out of the box' Blacklight SOLRmarc >>> config >>> to change it like so, would you guys agree? Not sure if that patch >>> goes >>> to Blacklight project or SolrMARC project? >>> >>> Except now I'm confused as to what solrmarc.properties files is >>> actually >>> included with the Blacklight demo. The "vanilla blacklight demo" >>> properties file in SolrMARC actually has: >>> >>> title_t = custom, removeTrailingPunct(245a) >>> >>> Is it possible to have >>> >>> title_t = custom, removeTrailingPunct(245aa), first ?? >>> >>> Should the properties file you get from a demo checkout be the >>> SolrMarc >>> "vanilla blacklight" example? I'm not sure what it is now...? >>> >>> Also, I could try to enhance the documentation on SolrMarc config to >>> explain what "first" does (currently in there by example, but not >>> actually described; are any other parameters valid there other than >>> 'first'?), and to explain that you can double-up a subfield to mean >>> "concatenate all such subfields from a given field" -- but I'm not >>> sure >>> I understand it well enough to write those docs? But maybe I'll >>> just go >>> go add that as I do understand it to the wiki? >>> http://code.google.com/p/solrmarc/wiki/ConfiguringSolrMarc I >>> guess I'd >>> need to get added to the SolrMarc project to have permission to edit >>> that wiki though. If anyone would like me to help enhance those >>> docs, >>> I'm _happy_ to do so, if someone can get me access to do so. >>> >>> Jonathan >>> >>> Robert Haschart wrote: >>> >>> >>>> Jonathan, >>>> >>>> Rather than changing the schema to not specify multiValued, you also >>>> have a number of other options: >>>> >>>> change the spec for getting the title value from: >>>> >>>> title_t = 245a >>>> >>>> to : >>>> >>>> title_t = 245a, first >>>> >>>> which will get only the first occurance of a given field/subfield. >>>> >>>> or to: >>>> >>>> title_t = 245aa >>>> >>>> which will concatenate all 'a' subfields for a given 245 field and >>>> return them as a single entry. >>>> >>>> or even: >>>> >>>> title_t = 245aa, first >>>> >>>> which will handle instances with multiple 'a' subfields in a >>>> single 245 >>>> field as above, but still not die if multiple 245 fields are >>>> present. >>>> >>>> Lastly two features that were added to Solrmarc very recently (as >>>> in the >>>> demo code does not yet have it) could help your situation. One is >>>> that >>>> errors such as missing required field, duplicate, non-multiValued >>>> fields, or field with unknown (to solr) names, will generate an >>>> error >>>> message for that record, but allow the indexing to continue. >>>> Second there is a new standard "custom" index function called >>>> getSingleIndexEntry which could be used like this: >>>> >>>> title_t = custom, getSingleIndexEntry(245aa, true) >>>> >>>> It would process the 245aa field spec as above, and then if multiple >>>> entries results it would select the longest result to use for the >>>> title_t index entry, and if the second parameter to the function >>>> is >>>> true (and if marc.include_errors is enabled) it will generate a >>>> marc_error index entry containing the additional errorneous 245 >>>> field >>>> data. >>>> >>>> -Bob Haschart >>>> >>>> >>>> Jonathan Rochkind wrote: >>>> >>>> >>>> >>>> >>>>> Cool. I'm not sure multiple 245$a's even technically _is_ "bad >>>>> data". >>>>> It's not neccesarily AACR2, but I found out the hard way recently >>>>> on >>>>> another project that we have LOTS of non-AACR2 data in our catalog, >>>>> including AACR1 data, pre-AACR data, and data cataloged to rare >>>>> books >>>>> and manuscripts standards, none of which our catalogers actually >>>>> consider 'bad'. >>>>> So Ross suggests one answer, changing it to: >>>>> >>>>> title_t = custom, removeTrailingPunct(245a), first >>>>> >>>>> So it'll ignore the second 245. Another option might be figuring >>>>> out >>>>> how to set the solrmarc setup to concatenate two 245$a's, rather >>>>> than >>>>> ignore the second one, which would seem to me to be the actually >>>>> appropriate thing to do in this case, barring letting title_t take >>>>> multiple values. Is it possible to do that somehow? >>>>> >>>>> Anyone have an opinion on if either of these two things should be >>>>> done >>>>> in the standard out-of-the-box demo setup, to accomodate this >>>>> kind of >>>>> data? >>>>> >>>>> Jonathan >>>>> >>>>> >>>>> >>>>> Naomi Dushay wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> Jonathan, >>>>>> >>>>>> We have a TON of bad data. Lots of records with multiple 245a; >>>>>> lots >>>>>> of records with other similar problems. >>>>>> >>>>>> We use solrmarc, and it nicely steps around these problems. >>>>>> There are >>>>>> solrmarc lists; >>>>>> >>>>>> solrmarc-general at googlegroups.com >>>>>> solrmarc-technical at googlegroups.com >>>>>> >>>>>> - Naomi >>>>>> >>>>>> >>>>>> On Jun 22, 2009, at 1:44 PM, Jonathan Rochkind wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Trying to take one baby step at a time in getting a little demo >>>>>>> of >>>>>>> Blacklight with our demo, I'm trying to index a sample of our own >>>>>>> local MARC records. >>>>>>> >>>>>>> I get an error from "rake app:index:marc", not sure exactly why, >>>>>>> error below. Maybe errors in my MARC? There definitely _will_ be >>>>>>> illegalities in my marc, would rather the indexer recovered from >>>>>>> them somehow (or just skipped that record, if nothing else is >>>>>>> possible), rather than punted the entire input. >>>>>>> >>>>>>> Should I switch to SolrMARC instead, is it a bit more >>>>>>> forgiving? At >>>>>>> one point I know I saw a pointer in the documentation to SolrMarc >>>>>>> docs, to help you get started with solrmarc instead of the >>>>>>> bundled >>>>>>> ruby indexer... but now I can't seem to find it. >>>>>>> >>>>>>> Here's what "raek app:index:marc" is telling me (giving me an >>>>>>> html >>>>>>> error message even though it's a command-line rake task, which >>>>>>> is a >>>>>>> bit odd). >>>>>>> rake aborted! >>>>>>> >>>>>>> >>>>>>> >>>>>>> Error 400 >>>>>>> >>>>>>>

HTTP ERROR: 400

ERROR: [53567] multiple values
>>>>>>> encountered for non multiValued field title_t: [Honan's
>>>>>>> Handbook to
>>>>>>> medical Europe, A ready reference book to the universities,
>>>>>>> hospitals, clinics, laboratories and general medical work of the
>>>>>>> principal cities of Europe]
>>>>>>>

RequestURI=/solr/update

>>>>>> href="http://jetty.mortbay.org/ >>>>>>> ">Powered by Jetty://


>>>>>>> _______________________________________________ >>>>>>> Blacklight-development mailing list >>>>>>> Blacklight-development at rubyforge.org >>>>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Blacklight-development mailing list >>>>>> Blacklight-development at rubyforge.org >>>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>>> >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Blacklight-development mailing list >>>>> Blacklight-development at rubyforge.org >>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From jamie at dang.com Tue Jun 23 11:33:05 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Tue, 23 Jun 2009 11:33:05 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <4A40F508.30400@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> Message-ID: These are using solr_marc. I think we need to get rid of those other tasks, unless Bess/Matt/Naomi say otherwise. Your change suggested in your previous email sounds like a good one. One thing I didn't like about the way I wrote the task was that you would always need to set the env variables when indexing for your app. On Jun 23, 2009, at 11:30 AM, Jonathan Rochkind wrote: > Oh, wait a second. The index command I was using right now is 'rake > app:index:marc'. I got that from the documentation. Should the > documentation be changed to reccommend "rake solr:marc:index" instead? > That documentation is actually in the source. I'll file a patch for > that too; I already filed a couple documentation patches, not sure > anyone is paying attention to them? > > Jamie Orchard-Hays wrote: >> rake solr:marc:index:info >> >> That will give you info you need to setup your config for the >> solrmarc >> indexer >> >> rake solr:marc:index >> >> is the actual command. >> >> There are two or three environment variables that the info task >> should >> indicate. >> >> We should get rid of rake app:index:ead and rake app:index:marc if >> they are not useful any longer. >> >> Jamie >> On Jun 23, 2009, at 10:11 AM, Jonathan Rochkind wrote: >> >> >>> So, now I'm a bit confused about where I change the indexing >>> configuration for SolrMarc in the demo app. >>> >>> The likely file I find that I think is controlling solrmarc when I >>> run a "rake app:index:marc" is at: >>> >>> bl-demo/rails/vendor/plugins/blacklight/config/demo_index.properties >>> >>> But it doesn't seem right to me to have to change a file in the >>> plugin source itself -- I want to leave the plugin pristine, so it >>> can be easily updated when new versions come out, and make changes >>> somewhere in my local app, right? >>> >>> What's the "right" way to make changes to the SolrMarc >>> configuration, such that when I run "rake app:index:marc" in the >>> demo app, they'll be used? >>> >>> Thanks for any help. Still curious if anyone has feedback on the >>> below post about whether the 'standard' demo app solrmarc config >>> should be changed? >>> >>> Jonathan >>> >>> >>> Jonathan Rochkind wrote: >>> >>>> Sweet, thanks a lot Robert, that's just what I needed to know. >>>> >>>> I believe that the 'right' thing to do, then, is: >>>> >>>> title_t = 245aa, first >>>> >>>> >>>> In the cases you were previously expecting, where there is only one >>>> 245$a, it will be indexed as before. In cases where there are >>>> multiple >>>> 245's or multiple $a's, then all of the "a"'s from the first 245 >>>> will be >>>> concatenated, and subsequent 245's will be ignored. I believe this >>>> is a >>>> better way to recover from that unexpected data than crashing >>>> entirely. >>>> >>>> If I submit a patch to the 'out of the box' Blacklight SOLRmarc >>>> config >>>> to change it like so, would you guys agree? Not sure if that patch >>>> goes >>>> to Blacklight project or SolrMARC project? >>>> >>>> Except now I'm confused as to what solrmarc.properties files is >>>> actually >>>> included with the Blacklight demo. The "vanilla blacklight demo" >>>> properties file in SolrMARC actually has: >>>> >>>> title_t = custom, removeTrailingPunct(245a) >>>> >>>> Is it possible to have >>>> >>>> title_t = custom, removeTrailingPunct(245aa), first ?? >>>> >>>> Should the properties file you get from a demo checkout be the >>>> SolrMarc >>>> "vanilla blacklight" example? I'm not sure what it is now...? >>>> >>>> Also, I could try to enhance the documentation on SolrMarc config >>>> to >>>> explain what "first" does (currently in there by example, but not >>>> actually described; are any other parameters valid there other than >>>> 'first'?), and to explain that you can double-up a subfield to mean >>>> "concatenate all such subfields from a given field" -- but I'm not >>>> sure >>>> I understand it well enough to write those docs? But maybe I'll >>>> just go >>>> go add that as I do understand it to the wiki? >>>> http://code.google.com/p/solrmarc/wiki/ConfiguringSolrMarc I >>>> guess I'd >>>> need to get added to the SolrMarc project to have permission to >>>> edit >>>> that wiki though. If anyone would like me to help enhance those >>>> docs, >>>> I'm _happy_ to do so, if someone can get me access to do so. >>>> >>>> Jonathan >>>> >>>> Robert Haschart wrote: >>>> >>>> >>>>> Jonathan, >>>>> >>>>> Rather than changing the schema to not specify multiValued, you >>>>> also >>>>> have a number of other options: >>>>> >>>>> change the spec for getting the title value from: >>>>> >>>>> title_t = 245a >>>>> >>>>> to : >>>>> >>>>> title_t = 245a, first >>>>> >>>>> which will get only the first occurance of a given field/subfield. >>>>> >>>>> or to: >>>>> >>>>> title_t = 245aa >>>>> >>>>> which will concatenate all 'a' subfields for a given 245 field >>>>> and >>>>> return them as a single entry. >>>>> >>>>> or even: >>>>> >>>>> title_t = 245aa, first >>>>> >>>>> which will handle instances with multiple 'a' subfields in a >>>>> single 245 >>>>> field as above, but still not die if multiple 245 fields are >>>>> present. >>>>> >>>>> Lastly two features that were added to Solrmarc very recently (as >>>>> in the >>>>> demo code does not yet have it) could help your situation. One is >>>>> that >>>>> errors such as missing required field, duplicate, non-multiValued >>>>> fields, or field with unknown (to solr) names, will generate an >>>>> error >>>>> message for that record, but allow the indexing to continue. >>>>> Second there is a new standard "custom" index function called >>>>> getSingleIndexEntry which could be used like this: >>>>> >>>>> title_t = custom, getSingleIndexEntry(245aa, true) >>>>> >>>>> It would process the 245aa field spec as above, and then if >>>>> multiple >>>>> entries results it would select the longest result to use for the >>>>> title_t index entry, and if the second parameter to the function >>>>> is >>>>> true (and if marc.include_errors is enabled) it will generate a >>>>> marc_error index entry containing the additional errorneous 245 >>>>> field >>>>> data. >>>>> >>>>> -Bob Haschart >>>>> >>>>> >>>>> Jonathan Rochkind wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> Cool. I'm not sure multiple 245$a's even technically _is_ "bad >>>>>> data". >>>>>> It's not neccesarily AACR2, but I found out the hard way recently >>>>>> on >>>>>> another project that we have LOTS of non-AACR2 data in our >>>>>> catalog, >>>>>> including AACR1 data, pre-AACR data, and data cataloged to rare >>>>>> books >>>>>> and manuscripts standards, none of which our catalogers actually >>>>>> consider 'bad'. >>>>>> So Ross suggests one answer, changing it to: >>>>>> >>>>>> title_t = custom, removeTrailingPunct(245a), first >>>>>> >>>>>> So it'll ignore the second 245. Another option might be figuring >>>>>> out >>>>>> how to set the solrmarc setup to concatenate two 245$a's, rather >>>>>> than >>>>>> ignore the second one, which would seem to me to be the actually >>>>>> appropriate thing to do in this case, barring letting title_t >>>>>> take >>>>>> multiple values. Is it possible to do that somehow? >>>>>> >>>>>> Anyone have an opinion on if either of these two things should be >>>>>> done >>>>>> in the standard out-of-the-box demo setup, to accomodate this >>>>>> kind of >>>>>> data? >>>>>> >>>>>> Jonathan >>>>>> >>>>>> >>>>>> >>>>>> Naomi Dushay wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Jonathan, >>>>>>> >>>>>>> We have a TON of bad data. Lots of records with multiple 245a; >>>>>>> lots >>>>>>> of records with other similar problems. >>>>>>> >>>>>>> We use solrmarc, and it nicely steps around these problems. >>>>>>> There are >>>>>>> solrmarc lists; >>>>>>> >>>>>>> solrmarc-general at googlegroups.com >>>>>>> solrmarc-technical at googlegroups.com >>>>>>> >>>>>>> - Naomi >>>>>>> >>>>>>> >>>>>>> On Jun 22, 2009, at 1:44 PM, Jonathan Rochkind wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Trying to take one baby step at a time in getting a little demo >>>>>>>> of >>>>>>>> Blacklight with our demo, I'm trying to index a sample of our >>>>>>>> own >>>>>>>> local MARC records. >>>>>>>> >>>>>>>> I get an error from "rake app:index:marc", not sure exactly >>>>>>>> why, >>>>>>>> error below. Maybe errors in my MARC? There definitely _will_ >>>>>>>> be >>>>>>>> illegalities in my marc, would rather the indexer recovered >>>>>>>> from >>>>>>>> them somehow (or just skipped that record, if nothing else is >>>>>>>> possible), rather than punted the entire input. >>>>>>>> >>>>>>>> Should I switch to SolrMARC instead, is it a bit more >>>>>>>> forgiving? At >>>>>>>> one point I know I saw a pointer in the documentation to >>>>>>>> SolrMarc >>>>>>>> docs, to help you get started with solrmarc instead of the >>>>>>>> bundled >>>>>>>> ruby indexer... but now I can't seem to find it. >>>>>>>> >>>>>>>> Here's what "raek app:index:marc" is telling me (giving me an >>>>>>>> html >>>>>>>> error message even though it's a command-line rake task, which >>>>>>>> is a >>>>>>>> bit odd). >>>>>>>> rake aborted! >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Error 400 >>>>>>>> >>>>>>>>

HTTP ERROR: 400

ERROR: [53567] multiple  
>>>>>>>> values
>>>>>>>> encountered for non multiValued field title_t: [Honan's
>>>>>>>> Handbook to
>>>>>>>> medical Europe, A ready reference book to the universities,
>>>>>>>> hospitals, clinics, laboratories and general medical work of  
>>>>>>>> the
>>>>>>>> principal cities of Europe]
>>>>>>>>

RequestURI=/solr/update

>>>>>>> href="http://jetty.mortbay.org/ >>>>>>>> ">Powered by Jetty://


>>>>>>>> _______________________________________________ >>>>>>>> Blacklight-development mailing list >>>>>>>> Blacklight-development at rubyforge.org >>>>>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Blacklight-development mailing list >>>>>>> Blacklight-development at rubyforge.org >>>>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Blacklight-development mailing list >>>>>> Blacklight-development at rubyforge.org >>>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Blacklight-development mailing list >>>>> Blacklight-development at rubyforge.org >>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From jamie at dang.com Tue Jun 23 11:35:42 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Tue, 23 Jun 2009 11:35:42 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <4A40EF4D.9010201@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <23b83f160906230720h4485aac2r2fe24acdf05a7b73@mail.gmail.com> <9B55AB03-FE07-4BF4-B402-EDB22BF4C332@virginia.edu> <4A40EF4D.9010201@jhu.edu> Message-ID: <670A1648-FA60-4F0A-BABC-B5DA7F3A555D@dang.com> Just remember that this has to be able to run without a surrounding rails app. The plugin also runs as its own app for testing purposes. On Jun 23, 2009, at 11:05 AM, Jonathan Rochkind wrote: > Yes, that does help, thanks, that makes sense. It looks like the > 'stock' solr_marc.rake task lives in the plugin (/lib/tasks/ > solr_marc.rake). I guess UVa and Stanford aren't actually using > that 'stock' rake task? > > Should I add instructions to a wiki page somewhere on this? > > As a more medium-term solution, if I modified the solr_marc.rake > task that lives in the plugin to take some configuration options > from the app instead, so the rake task itself doesn't need to be > cloned (and if it's later updated in the distro one can easily get > those updates) ... would the BL community be amenable to that patch? > Jonathan From rochkind at jhu.edu Tue Jun 23 11:37:27 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 23 Jun 2009 11:37:27 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <4A40F508.30400@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> Message-ID: <4A40F6B7.6050902@jhu.edu> Heh, but the defaults for solr:marc:index are in general not what's in the demo app. rake solr:marc:index MARC_RECORDS_PATH=../data/lc_records.utf8.mrc INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. Error: Supplied Solr home directory does not exist: ../../../../jetty/solr I'm actually not sure what arg to give solr:marc:index to fix where it's looking for Solr home directory. But I can poke around in the code and figure it out. It seems to me that the defaults for solr:marc:index should match what you get in the demo app though, yes? This can be changed by changing the defaults, OR by changing the setup of the 'demo' app. Which would you prefer? (I'd lean toward changing the defaults, because once you've checked out the demo app and put it into your local svn, you aren't going to get any future changes to it; but there might be a reason the defaults are what they are currently). I am more than happy to make these changes, submit patches, and write documentation. Just need guidance on what way you guys prefer it to go, so I don't spend time doing something you don't like and don't accept. Jonathan Jonathan Rochkind wrote: > Oh, wait a second. The index command I was using right now is 'rake > app:index:marc'. I got that from the documentation. Should the > documentation be changed to reccommend "rake solr:marc:index" instead? > > That documentation is actually in the source. I'll file a patch for that > too; I already filed a couple documentation patches, not sure anyone is > paying attention to them? > > Jamie Orchard-Hays wrote: > >> rake solr:marc:index:info >> >> That will give you info you need to setup your config for the solrmarc >> indexer >> >> rake solr:marc:index >> >> is the actual command. >> >> There are two or three environment variables that the info task should >> indicate. >> >> We should get rid of rake app:index:ead and rake app:index:marc if >> they are not useful any longer. >> >> Jamie >> On Jun 23, 2009, at 10:11 AM, Jonathan Rochkind wrote: >> >> >> >>> So, now I'm a bit confused about where I change the indexing >>> configuration for SolrMarc in the demo app. >>> >>> The likely file I find that I think is controlling solrmarc when I >>> run a "rake app:index:marc" is at: >>> >>> bl-demo/rails/vendor/plugins/blacklight/config/demo_index.properties >>> >>> But it doesn't seem right to me to have to change a file in the >>> plugin source itself -- I want to leave the plugin pristine, so it >>> can be easily updated when new versions come out, and make changes >>> somewhere in my local app, right? >>> >>> What's the "right" way to make changes to the SolrMarc >>> configuration, such that when I run "rake app:index:marc" in the >>> demo app, they'll be used? >>> >>> Thanks for any help. Still curious if anyone has feedback on the >>> below post about whether the 'standard' demo app solrmarc config >>> should be changed? >>> >>> Jonathan >>> >>> >>> Jonathan Rochkind wrote: >>> >>> >>>> Sweet, thanks a lot Robert, that's just what I needed to know. >>>> >>>> I believe that the 'right' thing to do, then, is: >>>> >>>> title_t = 245aa, first >>>> >>>> >>>> In the cases you were previously expecting, where there is only one >>>> 245$a, it will be indexed as before. In cases where there are >>>> multiple >>>> 245's or multiple $a's, then all of the "a"'s from the first 245 >>>> will be >>>> concatenated, and subsequent 245's will be ignored. I believe this >>>> is a >>>> better way to recover from that unexpected data than crashing >>>> entirely. >>>> >>>> If I submit a patch to the 'out of the box' Blacklight SOLRmarc >>>> config >>>> to change it like so, would you guys agree? Not sure if that patch >>>> goes >>>> to Blacklight project or SolrMARC project? >>>> >>>> Except now I'm confused as to what solrmarc.properties files is >>>> actually >>>> included with the Blacklight demo. The "vanilla blacklight demo" >>>> properties file in SolrMARC actually has: >>>> >>>> title_t = custom, removeTrailingPunct(245a) >>>> >>>> Is it possible to have >>>> >>>> title_t = custom, removeTrailingPunct(245aa), first ?? >>>> >>>> Should the properties file you get from a demo checkout be the >>>> SolrMarc >>>> "vanilla blacklight" example? I'm not sure what it is now...? >>>> >>>> Also, I could try to enhance the documentation on SolrMarc config to >>>> explain what "first" does (currently in there by example, but not >>>> actually described; are any other parameters valid there other than >>>> 'first'?), and to explain that you can double-up a subfield to mean >>>> "concatenate all such subfields from a given field" -- but I'm not >>>> sure >>>> I understand it well enough to write those docs? But maybe I'll >>>> just go >>>> go add that as I do understand it to the wiki? >>>> http://code.google.com/p/solrmarc/wiki/ConfiguringSolrMarc I >>>> guess I'd >>>> need to get added to the SolrMarc project to have permission to edit >>>> that wiki though. If anyone would like me to help enhance those >>>> docs, >>>> I'm _happy_ to do so, if someone can get me access to do so. >>>> >>>> Jonathan >>>> >>>> Robert Haschart wrote: >>>> >>>> >>>> >>>>> Jonathan, >>>>> >>>>> Rather than changing the schema to not specify multiValued, you also >>>>> have a number of other options: >>>>> >>>>> change the spec for getting the title value from: >>>>> >>>>> title_t = 245a >>>>> >>>>> to : >>>>> >>>>> title_t = 245a, first >>>>> >>>>> which will get only the first occurance of a given field/subfield. >>>>> >>>>> or to: >>>>> >>>>> title_t = 245aa >>>>> >>>>> which will concatenate all 'a' subfields for a given 245 field and >>>>> return them as a single entry. >>>>> >>>>> or even: >>>>> >>>>> title_t = 245aa, first >>>>> >>>>> which will handle instances with multiple 'a' subfields in a >>>>> single 245 >>>>> field as above, but still not die if multiple 245 fields are >>>>> present. >>>>> >>>>> Lastly two features that were added to Solrmarc very recently (as >>>>> in the >>>>> demo code does not yet have it) could help your situation. One is >>>>> that >>>>> errors such as missing required field, duplicate, non-multiValued >>>>> fields, or field with unknown (to solr) names, will generate an >>>>> error >>>>> message for that record, but allow the indexing to continue. >>>>> Second there is a new standard "custom" index function called >>>>> getSingleIndexEntry which could be used like this: >>>>> >>>>> title_t = custom, getSingleIndexEntry(245aa, true) >>>>> >>>>> It would process the 245aa field spec as above, and then if multiple >>>>> entries results it would select the longest result to use for the >>>>> title_t index entry, and if the second parameter to the function >>>>> is >>>>> true (and if marc.include_errors is enabled) it will generate a >>>>> marc_error index entry containing the additional errorneous 245 >>>>> field >>>>> data. >>>>> >>>>> -Bob Haschart >>>>> >>>>> >>>>> Jonathan Rochkind wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Cool. I'm not sure multiple 245$a's even technically _is_ "bad >>>>>> data". >>>>>> It's not neccesarily AACR2, but I found out the hard way recently >>>>>> on >>>>>> another project that we have LOTS of non-AACR2 data in our catalog, >>>>>> including AACR1 data, pre-AACR data, and data cataloged to rare >>>>>> books >>>>>> and manuscripts standards, none of which our catalogers actually >>>>>> consider 'bad'. >>>>>> So Ross suggests one answer, changing it to: >>>>>> >>>>>> title_t = custom, removeTrailingPunct(245a), first >>>>>> >>>>>> So it'll ignore the second 245. Another option might be figuring >>>>>> out >>>>>> how to set the solrmarc setup to concatenate two 245$a's, rather >>>>>> than >>>>>> ignore the second one, which would seem to me to be the actually >>>>>> appropriate thing to do in this case, barring letting title_t take >>>>>> multiple values. Is it possible to do that somehow? >>>>>> >>>>>> Anyone have an opinion on if either of these two things should be >>>>>> done >>>>>> in the standard out-of-the-box demo setup, to accomodate this >>>>>> kind of >>>>>> data? >>>>>> >>>>>> Jonathan >>>>>> >>>>>> >>>>>> >>>>>> Naomi Dushay wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Jonathan, >>>>>>> >>>>>>> We have a TON of bad data. Lots of records with multiple 245a; >>>>>>> lots >>>>>>> of records with other similar problems. >>>>>>> >>>>>>> We use solrmarc, and it nicely steps around these problems. >>>>>>> There are >>>>>>> solrmarc lists; >>>>>>> >>>>>>> solrmarc-general at googlegroups.com >>>>>>> solrmarc-technical at googlegroups.com >>>>>>> >>>>>>> - Naomi >>>>>>> >>>>>>> >>>>>>> On Jun 22, 2009, at 1:44 PM, Jonathan Rochkind wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Trying to take one baby step at a time in getting a little demo >>>>>>>> of >>>>>>>> Blacklight with our demo, I'm trying to index a sample of our own >>>>>>>> local MARC records. >>>>>>>> >>>>>>>> I get an error from "rake app:index:marc", not sure exactly why, >>>>>>>> error below. Maybe errors in my MARC? There definitely _will_ be >>>>>>>> illegalities in my marc, would rather the indexer recovered from >>>>>>>> them somehow (or just skipped that record, if nothing else is >>>>>>>> possible), rather than punted the entire input. >>>>>>>> >>>>>>>> Should I switch to SolrMARC instead, is it a bit more >>>>>>>> forgiving? At >>>>>>>> one point I know I saw a pointer in the documentation to SolrMarc >>>>>>>> docs, to help you get started with solrmarc instead of the >>>>>>>> bundled >>>>>>>> ruby indexer... but now I can't seem to find it. >>>>>>>> >>>>>>>> Here's what "raek app:index:marc" is telling me (giving me an >>>>>>>> html >>>>>>>> error message even though it's a command-line rake task, which >>>>>>>> is a >>>>>>>> bit odd). >>>>>>>> rake aborted! >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Error 400 >>>>>>>> >>>>>>>>

HTTP ERROR: 400

ERROR: [53567] multiple values
>>>>>>>> encountered for non multiValued field title_t: [Honan's
>>>>>>>> Handbook to
>>>>>>>> medical Europe, A ready reference book to the universities,
>>>>>>>> hospitals, clinics, laboratories and general medical work of the
>>>>>>>> principal cities of Europe]
>>>>>>>>

RequestURI=/solr/update

>>>>>>> href="http://jetty.mortbay.org/ >>>>>>>> ">Powered by Jetty://


>>>>>>>> _______________________________________________ >>>>>>>> Blacklight-development mailing list >>>>>>>> Blacklight-development at rubyforge.org >>>>>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Blacklight-development mailing list >>>>>>> Blacklight-development at rubyforge.org >>>>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Blacklight-development mailing list >>>>>> Blacklight-development at rubyforge.org >>>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>>> >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Blacklight-development mailing list >>>>> Blacklight-development at rubyforge.org >>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>> >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From eos8d at virginia.edu Tue Jun 23 11:40:05 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Tue, 23 Jun 2009 11:40:05 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> Message-ID: On 23-Jun-09, at 11:33 AM, Jamie Orchard-Hays wrote: > These are using solr_marc. I think we need to get rid of those other > tasks, unless Bess/Matt/Naomi say otherwise. We shouldn't ditch the EAD task, it's part of the EAD setup we're working on, and it will soon be joined by app:index:DC and app:index:MODS. But yeah, we shouldn't have two marc indexing tasks floating around, it's confusing. Bess From jamie at dang.com Tue Jun 23 11:45:58 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Tue, 23 Jun 2009 11:45:58 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <4A40F6B7.6050902@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> <4A40F6B7.6050902@jhu.edu> Message-ID: <04413ADA-50F3-4C9E-956F-29A6FD7704FE@dang.com> Sigh. You're right. I did that because I wanted some default in there and used the demo app locations. You all may need to revisit it. Unfortunately, I'm not actively on this project right now. Jamie On Jun 23, 2009, at 11:37 AM, Jonathan Rochkind wrote: > Heh, but the defaults for solr:marc:index are in general not what's > in the demo app. > > rake solr:marc:index MARC_RECORDS_PATH=../data/lc_records.utf8.mrc > INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. > Error: Supplied Solr home directory does not exist: ../../../../ > jetty/solr > > I'm actually not sure what arg to give solr:marc:index to fix where > it's looking for Solr home directory. But I can poke around in the > code and figure it out. > > It seems to me that the defaults for solr:marc:index should match > what you get in the demo app though, yes? This can be changed by > changing the defaults, OR by changing the setup of the 'demo' app. > Which would you prefer? (I'd lean toward changing the defaults, > because once you've checked out the demo app and put it into your > local svn, you aren't going to get any future changes to it; but > there might be a reason the defaults are what they are currently). > > I am more than happy to make these changes, submit patches, and > write documentation. Just need guidance on what way you guys prefer > it to go, so I don't spend time doing something you don't like and > don't accept. > > Jonathan From rochkind at jhu.edu Tue Jun 23 12:28:26 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 23 Jun 2009 12:28:26 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <04413ADA-50F3-4C9E-956F-29A6FD7704FE@dang.com> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> <4A40F6B7.6050902@jhu.edu> <04413ADA-50F3-4C9E-956F-29A6FD7704FE@dang.com> Message-ID: <4A4102AA.7060503@jhu.edu> Huh. But the default locations in there are NOT the (current) demo app locations. So it sounds like they were intended to be, and I can change them to be and submit a patch? I was going to ask Jamie if he wanted to have a chat in #blacklight on the right way to get the rake task working (in a trivial case) with the out-of-the-box demo app, but if he's not working on it right now... I'll just do my best that seems to make sense to me, and submit a patch? Jamie, there's no particular reason for the defaults chosen except that they are in fact supposed to match the demo app? I'll make it so. Jamie Orchard-Hays wrote: > Sigh. You're right. I did that because I wanted some default in there > and used the demo app locations. You all may need to revisit it. > Unfortunately, I'm not actively on this project right now. > > Jamie > > On Jun 23, 2009, at 11:37 AM, Jonathan Rochkind wrote: > > >> Heh, but the defaults for solr:marc:index are in general not what's >> in the demo app. >> >> rake solr:marc:index MARC_RECORDS_PATH=../data/lc_records.utf8.mrc >> INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. >> Error: Supplied Solr home directory does not exist: ../../../../ >> jetty/solr >> >> I'm actually not sure what arg to give solr:marc:index to fix where >> it's looking for Solr home directory. But I can poke around in the >> code and figure it out. >> >> It seems to me that the defaults for solr:marc:index should match >> what you get in the demo app though, yes? This can be changed by >> changing the defaults, OR by changing the setup of the 'demo' app. >> Which would you prefer? (I'd lean toward changing the defaults, >> because once you've checked out the demo app and put it into your >> local svn, you aren't going to get any future changes to it; but >> there might be a reason the defaults are what they are currently). >> >> I am more than happy to make these changes, submit patches, and >> write documentation. Just need guidance on what way you guys prefer >> it to go, so I don't spend time doing something you don't like and >> don't accept. >> >> Jonathan >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From jamie at dang.com Tue Jun 23 12:47:29 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Tue, 23 Jun 2009 12:47:29 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <4A4102AA.7060503@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> <4A40F6B7.6050902@jhu.edu> <04413ADA-50F3-4C9E-956F-29A6FD7704FE@dang.com> <4A4102AA.7060503@jhu.edu> Message-ID: <9718835E-8079-4CAD-A477-E15E65C36F77@dang.com> These match the default demo app location: default_solr_war_path = File.join(base_path, "../../../../../../jetty/webapps/solr.war") default_marc_records_path = File.join(base_path, "../../../../../../data/lc_records.utf8.mrc") These are files in the plugin: default_config_path = File.join(base_path, "../../config/ demo_config.properties") solr_marc_jar_path = File.join(base_path, "../../solr_marc/ SolrMarc.jar") On Jun 23, 2009, at 12:28 PM, Jonathan Rochkind wrote: > Huh. But the default locations in there are NOT the (current) demo > app locations. So it sounds like they were intended to be, and I > can change them to be and submit a patch? > > I was going to ask Jamie if he wanted to have a chat in #blacklight > on the right way to get the rake task working (in a trivial case) > with the out-of-the-box demo app, but if he's not working on it > right now... > > I'll just do my best that seems to make sense to me, and submit a > patch? Jamie, there's no particular reason for the defaults chosen > except that they are in fact supposed to match the demo app? I'll > make it so. > > Jamie Orchard-Hays wrote: >> Sigh. You're right. I did that because I wanted some default in there >> and used the demo app locations. You all may need to revisit it. >> Unfortunately, I'm not actively on this project right now. >> >> Jamie >> >> On Jun 23, 2009, at 11:37 AM, Jonathan Rochkind wrote: >> >> >>> Heh, but the defaults for solr:marc:index are in general not what's >>> in the demo app. >>> >>> rake solr:marc:index MARC_RECORDS_PATH=../data/lc_records.utf8.mrc >>> INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. >>> Error: Supplied Solr home directory does not exist: ../../../../ >>> jetty/solr >>> >>> I'm actually not sure what arg to give solr:marc:index to fix where >>> it's looking for Solr home directory. But I can poke around in the >>> code and figure it out. >>> >>> It seems to me that the defaults for solr:marc:index should match >>> what you get in the demo app though, yes? This can be changed by >>> changing the defaults, OR by changing the setup of the 'demo' app. >>> Which would you prefer? (I'd lean toward changing the defaults, >>> because once you've checked out the demo app and put it into your >>> local svn, you aren't going to get any future changes to it; but >>> there might be a reason the defaults are what they are currently). >>> >>> I am more than happy to make these changes, submit patches, and >>> write documentation. Just need guidance on what way you guys prefer >>> it to go, so I don't spend time doing something you don't like and >>> don't accept. >>> >>> Jonathan >>> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Tue Jun 23 12:58:02 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 23 Jun 2009 12:58:02 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <9718835E-8079-4CAD-A477-E15E65C36F77@dang.com> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> <4A40F6B7.6050902@jhu.edu> <04413ADA-50F3-4C9E-956F-29A6FD7704FE@dang.com> <4A4102AA.7060503@jhu.edu> <9718835E-8079-4CAD-A477-E15E65C36F77@dang.com> Message-ID: <4A41099A.2080008@jhu.edu> Huh, so then any guess as to why I'm getting the error I'm getting when I try to run the rake task against the demo app? Maybe it's the the demo_config.properties file in the plugin that doesn't match the demo? rake solr:marc:index MARC_RECORDS_PATH=../data/lc_records.utf8.mrc >>> INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. >>> Error: Supplied Solr home directory does not exist: ../../../../ >>> jetty/solr Jamie Orchard-Hays wrote: > These match the default demo app location: > > default_solr_war_path = File.join(base_path, > "../../../../../../jetty/webapps/solr.war") > default_marc_records_path = File.join(base_path, > "../../../../../../data/lc_records.utf8.mrc") > > These are files in the plugin: > > default_config_path = File.join(base_path, "../../config/ > demo_config.properties") > solr_marc_jar_path = File.join(base_path, "../../solr_marc/ > SolrMarc.jar") > > > On Jun 23, 2009, at 12:28 PM, Jonathan Rochkind wrote: > > >> Huh. But the default locations in there are NOT the (current) demo >> app locations. So it sounds like they were intended to be, and I >> can change them to be and submit a patch? >> >> I was going to ask Jamie if he wanted to have a chat in #blacklight >> on the right way to get the rake task working (in a trivial case) >> with the out-of-the-box demo app, but if he's not working on it >> right now... >> >> I'll just do my best that seems to make sense to me, and submit a >> patch? Jamie, there's no particular reason for the defaults chosen >> except that they are in fact supposed to match the demo app? I'll >> make it so. >> >> Jamie Orchard-Hays wrote: >> >>> Sigh. You're right. I did that because I wanted some default in there >>> and used the demo app locations. You all may need to revisit it. >>> Unfortunately, I'm not actively on this project right now. >>> >>> Jamie >>> >>> On Jun 23, 2009, at 11:37 AM, Jonathan Rochkind wrote: >>> >>> >>> >>>> Heh, but the defaults for solr:marc:index are in general not what's >>>> in the demo app. >>>> >>>> rake solr:marc:index MARC_RECORDS_PATH=../data/lc_records.utf8.mrc >>>> INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. >>>> Error: Supplied Solr home directory does not exist: ../../../../ >>>> jetty/solr >>>> >>>> I'm actually not sure what arg to give solr:marc:index to fix where >>>> it's looking for Solr home directory. But I can poke around in the >>>> code and figure it out. >>>> >>>> It seems to me that the defaults for solr:marc:index should match >>>> what you get in the demo app though, yes? This can be changed by >>>> changing the defaults, OR by changing the setup of the 'demo' app. >>>> Which would you prefer? (I'd lean toward changing the defaults, >>>> because once you've checked out the demo app and put it into your >>>> local svn, you aren't going to get any future changes to it; but >>>> there might be a reason the defaults are what they are currently). >>>> >>>> I am more than happy to make these changes, submit patches, and >>>> write documentation. Just need guidance on what way you guys prefer >>>> it to go, so I don't spend time doing something you don't like and >>>> don't accept. >>>> >>>> Jonathan >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From ndushay at stanford.edu Tue Jun 23 12:59:17 2009 From: ndushay at stanford.edu (Naomi Dushay) Date: Tue, 23 Jun 2009 09:59:17 -0700 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <4A4102AA.7060503@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> <4A40F6B7.6050902@jhu.edu> <04413ADA-50F3-4C9E-956F-29A6FD7704FE@dang.com> <4A4102AA.7060503@jhu.edu> Message-ID: Hi Jonathan, Oy, this is less obvious that we would have hoped. More documentation is definitely indicated. The demo index is basically intended as 1. a way for the plugin to have standalone tests, some of which need a SOLR index to run. 2. an exemplar. It's the 2. that's sub-optimal. We use solrmarc completely separately from the blacklight code to generate our index. Then we set up rails/config/solr.yml to point to the SOLR instance with the index we created. To do anything non-trivial with solrmarc probably implies you want to create a separate build process from solrmarc. You can certainly base it on the demo app ... but that's a very simple index that doesn't exploit much of the MARC data richness. I have it in my queue to flesh out the demo index more, so it's a better exemplar for MARC data. But I haven't gotten to it yet. And your notes make it clear that we have to cope with BOTH intentions above ... perhaps separately, or at least clearly documented. My apologies for this being so ... messy. Your experience, while frustrating for you, is very helpful to the project. And ... my recommendation is that you may want to use solrmarc independently, as UVa and Stanford do. You might start with the simple SOLR index fields the demo app uses, but populated from your data. However, you are likely to add a lot more fields to your index. You can check out http://searchworks-test.stanford.edu to see what our current blacklight efforts look like. You'll immediately notice how far we've strayed from the demo index. And if you saw the complexity of our search boosting formulae ... oy! - Naomi On Jun 23, 2009, at 9:28 AM, Jonathan Rochkind wrote: > Huh. But the default locations in there are NOT the (current) demo > app locations. So it sounds like they were intended to be, and I > can change them to be and submit a patch? > > I was going to ask Jamie if he wanted to have a chat in #blacklight > on the right way to get the rake task working (in a trivial case) > with the out-of-the-box demo app, but if he's not working on it > right now... > > I'll just do my best that seems to make sense to me, and submit a > patch? Jamie, there's no particular reason for the defaults chosen > except that they are in fact supposed to match the demo app? I'll > make it so. > > Jamie Orchard-Hays wrote: >> Sigh. You're right. I did that because I wanted some default in there >> and used the demo app locations. You all may need to revisit it. >> Unfortunately, I'm not actively on this project right now. >> >> Jamie >> >> On Jun 23, 2009, at 11:37 AM, Jonathan Rochkind wrote: >> >> >>> Heh, but the defaults for solr:marc:index are in general not what's >>> in the demo app. >>> >>> rake solr:marc:index MARC_RECORDS_PATH=../data/lc_records.utf8.mrc >>> INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. >>> Error: Supplied Solr home directory does not exist: ../../../../ >>> jetty/solr >>> >>> I'm actually not sure what arg to give solr:marc:index to fix where >>> it's looking for Solr home directory. But I can poke around in the >>> code and figure it out. >>> >>> It seems to me that the defaults for solr:marc:index should match >>> what you get in the demo app though, yes? This can be changed by >>> changing the defaults, OR by changing the setup of the 'demo' app. >>> Which would you prefer? (I'd lean toward changing the defaults, >>> because once you've checked out the demo app and put it into your >>> local svn, you aren't going to get any future changes to it; but >>> there might be a reason the defaults are what they are currently). >>> >>> I am more than happy to make these changes, submit patches, and >>> write documentation. Just need guidance on what way you guys prefer >>> it to go, so I don't spend time doing something you don't like and >>> don't accept. >>> >>> Jonathan >>> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From jamie at dang.com Tue Jun 23 13:03:16 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Tue, 23 Jun 2009 13:03:16 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> <4A40F6B7.6050902@jhu.edu> <04413ADA-50F3-4C9E-956F-29A6FD7704FE@dang.com> <4A4102AA.7060503@jhu.edu> Message-ID: <6A0BC61C-5131-40DE-A284-18D22DA54D20@dang.com> Actually, the standalone tests do not use the demo index! They have their own index which is actually committed to the repo and exists inside the jetty dir in the plugin. On Jun 23, 2009, at 12:59 PM, Naomi Dushay wrote: > Hi Jonathan, > > Oy, this is less obvious that we would have hoped. More > documentation is definitely indicated. > > The demo index is basically intended as > > 1. a way for the plugin to have standalone tests, some of which > need a SOLR index to run. > 2. an exemplar. From rochkind at jhu.edu Tue Jun 23 13:03:42 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 23 Jun 2009 13:03:42 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> <4A40F6B7.6050902@jhu.edu> <04413ADA-50F3-4C9E-956F-29A6FD7704FE@dang.com> <4A4102AA.7060503@jhu.edu> Message-ID: <4A410AEE.3010605@jhu.edu> I'd really like to help get things working so a new user can just use the rake task included there out-of-the-box for a simple demo. Like a new user _can_ use "rake app:index:marc" right now, with the newly checked out demo app, which is what the current instructions say to use. But apparently that's not actually using SolrMarc, and people think that task should be removed? In which case, the instructions which right now suggest that after installing the demo app you should run "rake app:index:marc FILE=../data/lc_records.utf8.mrc", should be changed to suggest you run rake solr:marc:index instead, right? I really think it's good to have this available (and am happy to help make it so). But right now, if you try running sorl:marc:index, you get errors. I'd really like to help fix things so this isn't so, so the instructions can say "And kick off the rake task to index some test data: rake solr:marc:index MARC_RECORDS_PATH=../data/test_data.utf8.mrc". That would be good, right? Is it possible for me to help make it so? That's a much easier learning curve (for me and those that come after me) to be able to start like this, then to say "Now, to index some data, go figure out how to install and use SolrMarc." Jonathan Naomi Dushay wrote: > Hi Jonathan, > > Oy, this is less obvious that we would have hoped. More documentation > is definitely indicated. > > The demo index is basically intended as > > 1. a way for the plugin to have standalone tests, some of which need > a SOLR index to run. > 2. an exemplar. > > It's the 2. that's sub-optimal. > > We use solrmarc completely separately from the blacklight code to > generate our index. Then we set up rails/config/solr.yml to point to > the SOLR instance with the index we created. > > To do anything non-trivial with solrmarc probably implies you want to > create a separate build process from solrmarc. You can certainly base > it on the demo app ... but that's a very simple index that doesn't > exploit much of the MARC data richness. > > I have it in my queue to flesh out the demo index more, so it's a > better exemplar for MARC data. But I haven't gotten to it yet. And > your notes make it clear that we have to cope with BOTH intentions > above ... perhaps separately, or at least clearly documented. > > My apologies for this being so ... messy. Your experience, while > frustrating for you, is very helpful to the project. > > And ... my recommendation is that you may want to use solrmarc > independently, as UVa and Stanford do. You might start with the > simple SOLR index fields the demo app uses, but populated from your > data. However, you are likely to add a lot more fields to your index. > > You can check out http://searchworks-test.stanford.edu to see what > our current blacklight efforts look like. You'll immediately notice > how far we've strayed from the demo index. And if you saw the > complexity of our search boosting formulae ... oy! > > - Naomi > > On Jun 23, 2009, at 9:28 AM, Jonathan Rochkind wrote: > > >> Huh. But the default locations in there are NOT the (current) demo >> app locations. So it sounds like they were intended to be, and I >> can change them to be and submit a patch? >> >> I was going to ask Jamie if he wanted to have a chat in #blacklight >> on the right way to get the rake task working (in a trivial case) >> with the out-of-the-box demo app, but if he's not working on it >> right now... >> >> I'll just do my best that seems to make sense to me, and submit a >> patch? Jamie, there's no particular reason for the defaults chosen >> except that they are in fact supposed to match the demo app? I'll >> make it so. >> >> Jamie Orchard-Hays wrote: >> >>> Sigh. You're right. I did that because I wanted some default in there >>> and used the demo app locations. You all may need to revisit it. >>> Unfortunately, I'm not actively on this project right now. >>> >>> Jamie >>> >>> On Jun 23, 2009, at 11:37 AM, Jonathan Rochkind wrote: >>> >>> >>> >>>> Heh, but the defaults for solr:marc:index are in general not what's >>>> in the demo app. >>>> >>>> rake solr:marc:index MARC_RECORDS_PATH=../data/lc_records.utf8.mrc >>>> INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. >>>> Error: Supplied Solr home directory does not exist: ../../../../ >>>> jetty/solr >>>> >>>> I'm actually not sure what arg to give solr:marc:index to fix where >>>> it's looking for Solr home directory. But I can poke around in the >>>> code and figure it out. >>>> >>>> It seems to me that the defaults for solr:marc:index should match >>>> what you get in the demo app though, yes? This can be changed by >>>> changing the defaults, OR by changing the setup of the 'demo' app. >>>> Which would you prefer? (I'd lean toward changing the defaults, >>>> because once you've checked out the demo app and put it into your >>>> local svn, you aren't going to get any future changes to it; but >>>> there might be a reason the defaults are what they are currently). >>>> >>>> I am more than happy to make these changes, submit patches, and >>>> write documentation. Just need guidance on what way you guys prefer >>>> it to go, so I don't spend time doing something you don't like and >>>> don't accept. >>>> >>>> Jonathan >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From rochkind at jhu.edu Tue Jun 23 13:06:23 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 23 Jun 2009 13:06:23 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> <4A40F6B7.6050902@jhu.edu> <04413ADA-50F3-4C9E-956F-29A6FD7704FE@dang.com> <4A4102AA.7060503@jhu.edu> Message-ID: <4A410B8F.1040005@jhu.edu> But, okay, it occurs to me that we can seperate #1 and #2 a bit. I can do what Jamie agreed was a good idea, and make the solr:marc:index task look in the app itself for a .properties file first. Then we can put a .properties file IN the demo app that actually works out of the box with the demo app (unlike the properties file in the plugin, which perhaps has to be the way it is for tests, which conflicts with it working out of the box for the demo app?) Does THAT make sense? I could do that, and submit a patch. If anyone can figure out the right way to do it, I'll do it and submit a patch and write documentation. :) Naomi Dushay wrote: > Hi Jonathan, > > Oy, this is less obvious that we would have hoped. More documentation > is definitely indicated. > > The demo index is basically intended as > > 1. a way for the plugin to have standalone tests, some of which need > a SOLR index to run. > 2. an exemplar. > > It's the 2. that's sub-optimal. > > We use solrmarc completely separately from the blacklight code to > generate our index. Then we set up rails/config/solr.yml to point to > the SOLR instance with the index we created. > > To do anything non-trivial with solrmarc probably implies you want to > create a separate build process from solrmarc. You can certainly base > it on the demo app ... but that's a very simple index that doesn't > exploit much of the MARC data richness. > > I have it in my queue to flesh out the demo index more, so it's a > better exemplar for MARC data. But I haven't gotten to it yet. And > your notes make it clear that we have to cope with BOTH intentions > above ... perhaps separately, or at least clearly documented. > > My apologies for this being so ... messy. Your experience, while > frustrating for you, is very helpful to the project. > > And ... my recommendation is that you may want to use solrmarc > independently, as UVa and Stanford do. You might start with the > simple SOLR index fields the demo app uses, but populated from your > data. However, you are likely to add a lot more fields to your index. > > You can check out http://searchworks-test.stanford.edu to see what > our current blacklight efforts look like. You'll immediately notice > how far we've strayed from the demo index. And if you saw the > complexity of our search boosting formulae ... oy! > > - Naomi > > On Jun 23, 2009, at 9:28 AM, Jonathan Rochkind wrote: > > >> Huh. But the default locations in there are NOT the (current) demo >> app locations. So it sounds like they were intended to be, and I >> can change them to be and submit a patch? >> >> I was going to ask Jamie if he wanted to have a chat in #blacklight >> on the right way to get the rake task working (in a trivial case) >> with the out-of-the-box demo app, but if he's not working on it >> right now... >> >> I'll just do my best that seems to make sense to me, and submit a >> patch? Jamie, there's no particular reason for the defaults chosen >> except that they are in fact supposed to match the demo app? I'll >> make it so. >> >> Jamie Orchard-Hays wrote: >> >>> Sigh. You're right. I did that because I wanted some default in there >>> and used the demo app locations. You all may need to revisit it. >>> Unfortunately, I'm not actively on this project right now. >>> >>> Jamie >>> >>> On Jun 23, 2009, at 11:37 AM, Jonathan Rochkind wrote: >>> >>> >>> >>>> Heh, but the defaults for solr:marc:index are in general not what's >>>> in the demo app. >>>> >>>> rake solr:marc:index MARC_RECORDS_PATH=../data/lc_records.utf8.mrc >>>> INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. >>>> Error: Supplied Solr home directory does not exist: ../../../../ >>>> jetty/solr >>>> >>>> I'm actually not sure what arg to give solr:marc:index to fix where >>>> it's looking for Solr home directory. But I can poke around in the >>>> code and figure it out. >>>> >>>> It seems to me that the defaults for solr:marc:index should match >>>> what you get in the demo app though, yes? This can be changed by >>>> changing the defaults, OR by changing the setup of the 'demo' app. >>>> Which would you prefer? (I'd lean toward changing the defaults, >>>> because once you've checked out the demo app and put it into your >>>> local svn, you aren't going to get any future changes to it; but >>>> there might be a reason the defaults are what they are currently). >>>> >>>> I am more than happy to make these changes, submit patches, and >>>> write documentation. Just need guidance on what way you guys prefer >>>> it to go, so I don't spend time doing something you don't like and >>>> don't accept. >>>> >>>> Jonathan >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From jamie at dang.com Tue Jun 23 13:39:05 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Tue, 23 Jun 2009 13:39:05 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <4A410AEE.3010605@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> <4A40F6B7.6050902@jhu.edu> <04413ADA-50F3-4C9E-956F-29A6FD7704FE@dang.com> <4A4102AA.7060503@jhu.edu> <4A410AEE.3010605@jhu.edu> Message-ID: <8B11FE0B-4710-4188-A0C9-3772C632C816@dang.com> OK. I just downloaded 2.2, untarred and tried out rake solr:marc:index. If I'm in the rails dir, it fails due to errors (the path is wrong). If I'm in the blacklight plugin dir, it works, because the path is right. The way to fix it is to have it use the __FILE__ location to orient the path properly. Except, that IS what the rake task does. So somehow when rake loads up all the tasks, it's not referring to the actual __FILE__ the task is created in. This is not the behavior I expect from Ruby. My understanding is that it should use that particular file is the reference. I'm perplexed. On Jun 23, 2009, at 1:03 PM, Jonathan Rochkind wrote: > I'd really like to help get things working so a new user can just > use the rake task included there out-of-the-box for a simple demo. > > Like a new user _can_ use "rake app:index:marc" right now, with the > newly checked out demo app, which is what the current instructions > say to use. But apparently that's not actually using SolrMarc, and > people think that task should be removed? > > In which case, the instructions which right now suggest that after > installing the demo app you should run "rake app:index:marc FILE=../ > data/lc_records.utf8.mrc", should be changed to suggest you run rake > solr:marc:index instead, right? I really think it's good to have > this available (and am happy to help make it so). > > But right now, if you try running sorl:marc:index, you get errors. > I'd really like to help fix things so this isn't so, so the > instructions can say "And kick off the rake task to index some test > data: rake solr:marc:index MARC_RECORDS_PATH=../data/ > test_data.utf8.mrc". > > That would be good, right? Is it possible for me to help make it > so? That's a much easier learning curve (for me and those that come > after me) to be able to start like this, then to say "Now, to index > some data, go figure out how to install and use SolrMarc." > > Jonathan > > Naomi Dushay wrote: >> Hi Jonathan, >> >> Oy, this is less obvious that we would have hoped. More >> documentation >> is definitely indicated. >> >> The demo index is basically intended as >> >> 1. a way for the plugin to have standalone tests, some of which need >> a SOLR index to run. >> 2. an exemplar. >> >> It's the 2. that's sub-optimal. >> >> We use solrmarc completely separately from the blacklight code to >> generate our index. Then we set up rails/config/solr.yml to point >> to >> the SOLR instance with the index we created. >> >> To do anything non-trivial with solrmarc probably implies you want to >> create a separate build process from solrmarc. You can certainly >> base >> it on the demo app ... but that's a very simple index that doesn't >> exploit much of the MARC data richness. >> >> I have it in my queue to flesh out the demo index more, so it's a >> better exemplar for MARC data. But I haven't gotten to it yet. And >> your notes make it clear that we have to cope with BOTH intentions >> above ... perhaps separately, or at least clearly documented. >> >> My apologies for this being so ... messy. Your experience, while >> frustrating for you, is very helpful to the project. >> >> And ... my recommendation is that you may want to use solrmarc >> independently, as UVa and Stanford do. You might start with the >> simple SOLR index fields the demo app uses, but populated from your >> data. However, you are likely to add a lot more fields to your >> index. >> >> You can check out http://searchworks-test.stanford.edu to see what >> our current blacklight efforts look like. You'll immediately notice >> how far we've strayed from the demo index. And if you saw the >> complexity of our search boosting formulae ... oy! >> >> - Naomi >> >> On Jun 23, 2009, at 9:28 AM, Jonathan Rochkind wrote: >> >> >>> Huh. But the default locations in there are NOT the (current) demo >>> app locations. So it sounds like they were intended to be, and I >>> can change them to be and submit a patch? >>> >>> I was going to ask Jamie if he wanted to have a chat in #blacklight >>> on the right way to get the rake task working (in a trivial case) >>> with the out-of-the-box demo app, but if he's not working on it >>> right now... >>> >>> I'll just do my best that seems to make sense to me, and submit a >>> patch? Jamie, there's no particular reason for the defaults chosen >>> except that they are in fact supposed to match the demo app? I'll >>> make it so. >>> >>> Jamie Orchard-Hays wrote: >>> >>>> Sigh. You're right. I did that because I wanted some default in >>>> there >>>> and used the demo app locations. You all may need to revisit it. >>>> Unfortunately, I'm not actively on this project right now. >>>> >>>> Jamie >>>> >>>> On Jun 23, 2009, at 11:37 AM, Jonathan Rochkind wrote: >>>> >>>> >>>> >>>>> Heh, but the defaults for solr:marc:index are in general not >>>>> what's >>>>> in the demo app. >>>>> >>>>> rake solr:marc:index MARC_RECORDS_PATH=../data/lc_records.utf8.mrc >>>>> INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. >>>>> Error: Supplied Solr home directory does not exist: ../../../../ >>>>> jetty/solr >>>>> >>>>> I'm actually not sure what arg to give solr:marc:index to fix >>>>> where >>>>> it's looking for Solr home directory. But I can poke around in >>>>> the >>>>> code and figure it out. >>>>> >>>>> It seems to me that the defaults for solr:marc:index should match >>>>> what you get in the demo app though, yes? This can be changed by >>>>> changing the defaults, OR by changing the setup of the 'demo' app. >>>>> Which would you prefer? (I'd lean toward changing the defaults, >>>>> because once you've checked out the demo app and put it into your >>>>> local svn, you aren't going to get any future changes to it; but >>>>> there might be a reason the defaults are what they are currently). >>>>> >>>>> I am more than happy to make these changes, submit patches, and >>>>> write documentation. Just need guidance on what way you guys >>>>> prefer >>>>> it to go, so I don't spend time doing something you don't like and >>>>> don't accept. >>>>> >>>>> Jonathan >>>>> >>>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Tue Jun 23 13:50:16 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 23 Jun 2009 13:50:16 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <8B11FE0B-4710-4188-A0C9-3772C632C816@dang.com> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> <4A40F6B7.6050902@jhu.edu> <04413ADA-50F3-4C9E-956F-29A6FD7704FE@dang.com> <4A4102AA.7060503@jhu.edu> <4A410AEE.3010605@jhu.edu> <8B11FE0B-4710-4188-A0C9-3772C632C816@dang.com> Message-ID: <4A4115D8.7060906@jhu.edu> Sweet, that's very helpful. I'll work on debugging it myself tomorrow too, with those hints about what's going on. Jamie Orchard-Hays wrote: > OK. I just downloaded 2.2, untarred and tried out rake solr:marc:index. > > If I'm in the rails dir, it fails due to errors (the path is wrong). > > If I'm in the blacklight plugin dir, it works, because the path is > right. > > The way to fix it is to have it use the __FILE__ location to orient > the path properly. > > Except, that IS what the rake task does. So somehow when rake loads up > all the tasks, it's not referring to the actual __FILE__ the task is > created in. This is not the behavior I expect from Ruby. My > understanding is that it should use that particular file is the > reference. > > I'm perplexed. > > > On Jun 23, 2009, at 1:03 PM, Jonathan Rochkind wrote: > > >> I'd really like to help get things working so a new user can just >> use the rake task included there out-of-the-box for a simple demo. >> >> Like a new user _can_ use "rake app:index:marc" right now, with the >> newly checked out demo app, which is what the current instructions >> say to use. But apparently that's not actually using SolrMarc, and >> people think that task should be removed? >> >> In which case, the instructions which right now suggest that after >> installing the demo app you should run "rake app:index:marc FILE=../ >> data/lc_records.utf8.mrc", should be changed to suggest you run rake >> solr:marc:index instead, right? I really think it's good to have >> this available (and am happy to help make it so). >> >> But right now, if you try running sorl:marc:index, you get errors. >> I'd really like to help fix things so this isn't so, so the >> instructions can say "And kick off the rake task to index some test >> data: rake solr:marc:index MARC_RECORDS_PATH=../data/ >> test_data.utf8.mrc". >> >> That would be good, right? Is it possible for me to help make it >> so? That's a much easier learning curve (for me and those that come >> after me) to be able to start like this, then to say "Now, to index >> some data, go figure out how to install and use SolrMarc." >> >> Jonathan >> >> Naomi Dushay wrote: >> >>> Hi Jonathan, >>> >>> Oy, this is less obvious that we would have hoped. More >>> documentation >>> is definitely indicated. >>> >>> The demo index is basically intended as >>> >>> 1. a way for the plugin to have standalone tests, some of which need >>> a SOLR index to run. >>> 2. an exemplar. >>> >>> It's the 2. that's sub-optimal. >>> >>> We use solrmarc completely separately from the blacklight code to >>> generate our index. Then we set up rails/config/solr.yml to point >>> to >>> the SOLR instance with the index we created. >>> >>> To do anything non-trivial with solrmarc probably implies you want to >>> create a separate build process from solrmarc. You can certainly >>> base >>> it on the demo app ... but that's a very simple index that doesn't >>> exploit much of the MARC data richness. >>> >>> I have it in my queue to flesh out the demo index more, so it's a >>> better exemplar for MARC data. But I haven't gotten to it yet. And >>> your notes make it clear that we have to cope with BOTH intentions >>> above ... perhaps separately, or at least clearly documented. >>> >>> My apologies for this being so ... messy. Your experience, while >>> frustrating for you, is very helpful to the project. >>> >>> And ... my recommendation is that you may want to use solrmarc >>> independently, as UVa and Stanford do. You might start with the >>> simple SOLR index fields the demo app uses, but populated from your >>> data. However, you are likely to add a lot more fields to your >>> index. >>> >>> You can check out http://searchworks-test.stanford.edu to see what >>> our current blacklight efforts look like. You'll immediately notice >>> how far we've strayed from the demo index. And if you saw the >>> complexity of our search boosting formulae ... oy! >>> >>> - Naomi >>> >>> On Jun 23, 2009, at 9:28 AM, Jonathan Rochkind wrote: >>> >>> >>> >>>> Huh. But the default locations in there are NOT the (current) demo >>>> app locations. So it sounds like they were intended to be, and I >>>> can change them to be and submit a patch? >>>> >>>> I was going to ask Jamie if he wanted to have a chat in #blacklight >>>> on the right way to get the rake task working (in a trivial case) >>>> with the out-of-the-box demo app, but if he's not working on it >>>> right now... >>>> >>>> I'll just do my best that seems to make sense to me, and submit a >>>> patch? Jamie, there's no particular reason for the defaults chosen >>>> except that they are in fact supposed to match the demo app? I'll >>>> make it so. >>>> >>>> Jamie Orchard-Hays wrote: >>>> >>>> >>>>> Sigh. You're right. I did that because I wanted some default in >>>>> there >>>>> and used the demo app locations. You all may need to revisit it. >>>>> Unfortunately, I'm not actively on this project right now. >>>>> >>>>> Jamie >>>>> >>>>> On Jun 23, 2009, at 11:37 AM, Jonathan Rochkind wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> Heh, but the defaults for solr:marc:index are in general not >>>>>> what's >>>>>> in the demo app. >>>>>> >>>>>> rake solr:marc:index MARC_RECORDS_PATH=../data/lc_records.utf8.mrc >>>>>> INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. >>>>>> Error: Supplied Solr home directory does not exist: ../../../../ >>>>>> jetty/solr >>>>>> >>>>>> I'm actually not sure what arg to give solr:marc:index to fix >>>>>> where >>>>>> it's looking for Solr home directory. But I can poke around in >>>>>> the >>>>>> code and figure it out. >>>>>> >>>>>> It seems to me that the defaults for solr:marc:index should match >>>>>> what you get in the demo app though, yes? This can be changed by >>>>>> changing the defaults, OR by changing the setup of the 'demo' app. >>>>>> Which would you prefer? (I'd lean toward changing the defaults, >>>>>> because once you've checked out the demo app and put it into your >>>>>> local svn, you aren't going to get any future changes to it; but >>>>>> there might be a reason the defaults are what they are currently). >>>>>> >>>>>> I am more than happy to make these changes, submit patches, and >>>>>> write documentation. Just need guidance on what way you guys >>>>>> prefer >>>>>> it to go, so I don't spend time doing something you don't like and >>>>>> don't accept. >>>>>> >>>>>> Jonathan >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Blacklight-development mailing list >>>>> Blacklight-development at rubyforge.org >>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From rochkind at jhu.edu Tue Jun 23 13:58:50 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 23 Jun 2009 13:58:50 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <8B11FE0B-4710-4188-A0C9-3772C632C816@dang.com> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> <4A40F6B7.6050902@jhu.edu> <04413ADA-50F3-4C9E-956F-29A6FD7704FE@dang.com> <4A4102AA.7060503@jhu.edu> <4A410AEE.3010605@jhu.edu> <8B11FE0B-4710-4188-A0C9-3772C632C816@dang.com> Message-ID: <4A4117DA.4020202@jhu.edu> Is using RAILS_ROOT right out, because it'll mess things up when the plug-in is tested without an app? Can someone tell me how I run the plugin tests, so I can make sure anything I do isn't breaking them? Is it just "rake test", or is it different for the rspec tests you've got? I haven't used rspec before, just the built in testing. Jamie Orchard-Hays wrote: > OK. I just downloaded 2.2, untarred and tried out rake solr:marc:index. > > If I'm in the rails dir, it fails due to errors (the path is wrong). > > If I'm in the blacklight plugin dir, it works, because the path is > right. > > The way to fix it is to have it use the __FILE__ location to orient > the path properly. > > Except, that IS what the rake task does. So somehow when rake loads up > all the tasks, it's not referring to the actual __FILE__ the task is > created in. This is not the behavior I expect from Ruby. My > understanding is that it should use that particular file is the > reference. > > I'm perplexed. > > > On Jun 23, 2009, at 1:03 PM, Jonathan Rochkind wrote: > > >> I'd really like to help get things working so a new user can just >> use the rake task included there out-of-the-box for a simple demo. >> >> Like a new user _can_ use "rake app:index:marc" right now, with the >> newly checked out demo app, which is what the current instructions >> say to use. But apparently that's not actually using SolrMarc, and >> people think that task should be removed? >> >> In which case, the instructions which right now suggest that after >> installing the demo app you should run "rake app:index:marc FILE=../ >> data/lc_records.utf8.mrc", should be changed to suggest you run rake >> solr:marc:index instead, right? I really think it's good to have >> this available (and am happy to help make it so). >> >> But right now, if you try running sorl:marc:index, you get errors. >> I'd really like to help fix things so this isn't so, so the >> instructions can say "And kick off the rake task to index some test >> data: rake solr:marc:index MARC_RECORDS_PATH=../data/ >> test_data.utf8.mrc". >> >> That would be good, right? Is it possible for me to help make it >> so? That's a much easier learning curve (for me and those that come >> after me) to be able to start like this, then to say "Now, to index >> some data, go figure out how to install and use SolrMarc." >> >> Jonathan >> >> Naomi Dushay wrote: >> >>> Hi Jonathan, >>> >>> Oy, this is less obvious that we would have hoped. More >>> documentation >>> is definitely indicated. >>> >>> The demo index is basically intended as >>> >>> 1. a way for the plugin to have standalone tests, some of which need >>> a SOLR index to run. >>> 2. an exemplar. >>> >>> It's the 2. that's sub-optimal. >>> >>> We use solrmarc completely separately from the blacklight code to >>> generate our index. Then we set up rails/config/solr.yml to point >>> to >>> the SOLR instance with the index we created. >>> >>> To do anything non-trivial with solrmarc probably implies you want to >>> create a separate build process from solrmarc. You can certainly >>> base >>> it on the demo app ... but that's a very simple index that doesn't >>> exploit much of the MARC data richness. >>> >>> I have it in my queue to flesh out the demo index more, so it's a >>> better exemplar for MARC data. But I haven't gotten to it yet. And >>> your notes make it clear that we have to cope with BOTH intentions >>> above ... perhaps separately, or at least clearly documented. >>> >>> My apologies for this being so ... messy. Your experience, while >>> frustrating for you, is very helpful to the project. >>> >>> And ... my recommendation is that you may want to use solrmarc >>> independently, as UVa and Stanford do. You might start with the >>> simple SOLR index fields the demo app uses, but populated from your >>> data. However, you are likely to add a lot more fields to your >>> index. >>> >>> You can check out http://searchworks-test.stanford.edu to see what >>> our current blacklight efforts look like. You'll immediately notice >>> how far we've strayed from the demo index. And if you saw the >>> complexity of our search boosting formulae ... oy! >>> >>> - Naomi >>> >>> On Jun 23, 2009, at 9:28 AM, Jonathan Rochkind wrote: >>> >>> >>> >>>> Huh. But the default locations in there are NOT the (current) demo >>>> app locations. So it sounds like they were intended to be, and I >>>> can change them to be and submit a patch? >>>> >>>> I was going to ask Jamie if he wanted to have a chat in #blacklight >>>> on the right way to get the rake task working (in a trivial case) >>>> with the out-of-the-box demo app, but if he's not working on it >>>> right now... >>>> >>>> I'll just do my best that seems to make sense to me, and submit a >>>> patch? Jamie, there's no particular reason for the defaults chosen >>>> except that they are in fact supposed to match the demo app? I'll >>>> make it so. >>>> >>>> Jamie Orchard-Hays wrote: >>>> >>>> >>>>> Sigh. You're right. I did that because I wanted some default in >>>>> there >>>>> and used the demo app locations. You all may need to revisit it. >>>>> Unfortunately, I'm not actively on this project right now. >>>>> >>>>> Jamie >>>>> >>>>> On Jun 23, 2009, at 11:37 AM, Jonathan Rochkind wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> Heh, but the defaults for solr:marc:index are in general not >>>>>> what's >>>>>> in the demo app. >>>>>> >>>>>> rake solr:marc:index MARC_RECORDS_PATH=../data/lc_records.utf8.mrc >>>>>> INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. >>>>>> Error: Supplied Solr home directory does not exist: ../../../../ >>>>>> jetty/solr >>>>>> >>>>>> I'm actually not sure what arg to give solr:marc:index to fix >>>>>> where >>>>>> it's looking for Solr home directory. But I can poke around in >>>>>> the >>>>>> code and figure it out. >>>>>> >>>>>> It seems to me that the defaults for solr:marc:index should match >>>>>> what you get in the demo app though, yes? This can be changed by >>>>>> changing the defaults, OR by changing the setup of the 'demo' app. >>>>>> Which would you prefer? (I'd lean toward changing the defaults, >>>>>> because once you've checked out the demo app and put it into your >>>>>> local svn, you aren't going to get any future changes to it; but >>>>>> there might be a reason the defaults are what they are currently). >>>>>> >>>>>> I am more than happy to make these changes, submit patches, and >>>>>> write documentation. Just need guidance on what way you guys >>>>>> prefer >>>>>> it to go, so I don't spend time doing something you don't like and >>>>>> don't accept. >>>>>> >>>>>> Jonathan >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Blacklight-development mailing list >>>>> Blacklight-development at rubyforge.org >>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From eos8d at virginia.edu Tue Jun 23 15:09:54 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Tue, 23 Jun 2009 15:09:54 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <4A4117DA.4020202@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> <4A40F6B7.6050902@jhu.edu> <04413ADA-50F3-4C9E-956F-29A6FD7704FE@dang.com> <4A4102AA.7060503@jhu.edu> <4A410AEE.3010605@jhu.edu> <8B11FE0B-4710-4188-A0C9-3772C632C816@dang.com> <4A4117DA.4020202@jhu.edu> Message-ID: <44358295-7F59-4E2C-98A2-8924A37B694C@virginia.edu> On Jun 23, 2009, at 1:58 PM, Jonathan Rochkind wrote: > Can someone tell me how I run the plugin tests, so I can make sure > anything I do isn't breaking them? Is it just "rake test", or is it > different for the rspec tests you've got? I haven't used rspec before, > just the built in testing. cd vendor/plugins/blacklight rake -T solr (to see the list of tasks) rake solr:features rake solr:spec These will spin up a local copy of solr that has all the test data indexed, and run the tests against it. Bess From rochkind at jhu.edu Wed Jun 24 12:36:24 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Wed, 24 Jun 2009 12:36:24 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <8B11FE0B-4710-4188-A0C9-3772C632C816@dang.com> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> <4A40F6B7.6050902@jhu.edu> <04413ADA-50F3-4C9E-956F-29A6FD7704FE@dang.com> <4A4102AA.7060503@jhu.edu> <4A410AEE.3010605@jhu.edu> <8B11FE0B-4710-4188-A0C9-3772C632C816@dang.com> Message-ID: <4A425608.6090403@jhu.edu> Hey Jamie, so this is weird, I did get 'rake solr:marc:index' to work once when I tried 'cd'ing into the bl plugin directory first, as you advise. (Still thinking about/investigating how to fix it so this isn't necessary). But when I try it a second time, with other records (or with the same mrc record file), it doesn't work. With the below exception. Any ideas at all? I don't know why it would work once, but not twice. I don't think I've made any changes? Or do I need to go ask on the solrmarc list about this? [rochkind at xs001 blacklight]$ rake solr:marc:index MARC_RECORDS_PATH=../../../../data/lc_records.utf8.mrc (in /opt/blacklight/bl-demo/rails/vendor/plugins/blacklight) INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. Error: Problem instantiating SolrCore ERROR [main] (SolrCoreLoader.java:157) - Error: Problem instantiating SolrCore java.lang.reflect.InvocationTargetException Jamie Orchard-Hays wrote: > OK. I just downloaded 2.2, untarred and tried out rake solr:marc:index. > > If I'm in the rails dir, it fails due to errors (the path is wrong). > > If I'm in the blacklight plugin dir, it works, because the path is > right. > > The way to fix it is to have it use the __FILE__ location to orient > the path properly. > > Except, that IS what the rake task does. So somehow when rake loads up > all the tasks, it's not referring to the actual __FILE__ the task is > created in. This is not the behavior I expect from Ruby. My > understanding is that it should use that particular file is the > reference. > > I'm perplexed. > > > On Jun 23, 2009, at 1:03 PM, Jonathan Rochkind wrote: > > >> I'd really like to help get things working so a new user can just >> use the rake task included there out-of-the-box for a simple demo. >> >> Like a new user _can_ use "rake app:index:marc" right now, with the >> newly checked out demo app, which is what the current instructions >> say to use. But apparently that's not actually using SolrMarc, and >> people think that task should be removed? >> >> In which case, the instructions which right now suggest that after >> installing the demo app you should run "rake app:index:marc FILE=../ >> data/lc_records.utf8.mrc", should be changed to suggest you run rake >> solr:marc:index instead, right? I really think it's good to have >> this available (and am happy to help make it so). >> >> But right now, if you try running sorl:marc:index, you get errors. >> I'd really like to help fix things so this isn't so, so the >> instructions can say "And kick off the rake task to index some test >> data: rake solr:marc:index MARC_RECORDS_PATH=../data/ >> test_data.utf8.mrc". >> >> That would be good, right? Is it possible for me to help make it >> so? That's a much easier learning curve (for me and those that come >> after me) to be able to start like this, then to say "Now, to index >> some data, go figure out how to install and use SolrMarc." >> >> Jonathan >> >> Naomi Dushay wrote: >> >>> Hi Jonathan, >>> >>> Oy, this is less obvious that we would have hoped. More >>> documentation >>> is definitely indicated. >>> >>> The demo index is basically intended as >>> >>> 1. a way for the plugin to have standalone tests, some of which need >>> a SOLR index to run. >>> 2. an exemplar. >>> >>> It's the 2. that's sub-optimal. >>> >>> We use solrmarc completely separately from the blacklight code to >>> generate our index. Then we set up rails/config/solr.yml to point >>> to >>> the SOLR instance with the index we created. >>> >>> To do anything non-trivial with solrmarc probably implies you want to >>> create a separate build process from solrmarc. You can certainly >>> base >>> it on the demo app ... but that's a very simple index that doesn't >>> exploit much of the MARC data richness. >>> >>> I have it in my queue to flesh out the demo index more, so it's a >>> better exemplar for MARC data. But I haven't gotten to it yet. And >>> your notes make it clear that we have to cope with BOTH intentions >>> above ... perhaps separately, or at least clearly documented. >>> >>> My apologies for this being so ... messy. Your experience, while >>> frustrating for you, is very helpful to the project. >>> >>> And ... my recommendation is that you may want to use solrmarc >>> independently, as UVa and Stanford do. You might start with the >>> simple SOLR index fields the demo app uses, but populated from your >>> data. However, you are likely to add a lot more fields to your >>> index. >>> >>> You can check out http://searchworks-test.stanford.edu to see what >>> our current blacklight efforts look like. You'll immediately notice >>> how far we've strayed from the demo index. And if you saw the >>> complexity of our search boosting formulae ... oy! >>> >>> - Naomi >>> >>> On Jun 23, 2009, at 9:28 AM, Jonathan Rochkind wrote: >>> >>> >>> >>>> Huh. But the default locations in there are NOT the (current) demo >>>> app locations. So it sounds like they were intended to be, and I >>>> can change them to be and submit a patch? >>>> >>>> I was going to ask Jamie if he wanted to have a chat in #blacklight >>>> on the right way to get the rake task working (in a trivial case) >>>> with the out-of-the-box demo app, but if he's not working on it >>>> right now... >>>> >>>> I'll just do my best that seems to make sense to me, and submit a >>>> patch? Jamie, there's no particular reason for the defaults chosen >>>> except that they are in fact supposed to match the demo app? I'll >>>> make it so. >>>> >>>> Jamie Orchard-Hays wrote: >>>> >>>> >>>>> Sigh. You're right. I did that because I wanted some default in >>>>> there >>>>> and used the demo app locations. You all may need to revisit it. >>>>> Unfortunately, I'm not actively on this project right now. >>>>> >>>>> Jamie >>>>> >>>>> On Jun 23, 2009, at 11:37 AM, Jonathan Rochkind wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> Heh, but the defaults for solr:marc:index are in general not >>>>>> what's >>>>>> in the demo app. >>>>>> >>>>>> rake solr:marc:index MARC_RECORDS_PATH=../data/lc_records.utf8.mrc >>>>>> INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. >>>>>> Error: Supplied Solr home directory does not exist: ../../../../ >>>>>> jetty/solr >>>>>> >>>>>> I'm actually not sure what arg to give solr:marc:index to fix >>>>>> where >>>>>> it's looking for Solr home directory. But I can poke around in >>>>>> the >>>>>> code and figure it out. >>>>>> >>>>>> It seems to me that the defaults for solr:marc:index should match >>>>>> what you get in the demo app though, yes? This can be changed by >>>>>> changing the defaults, OR by changing the setup of the 'demo' app. >>>>>> Which would you prefer? (I'd lean toward changing the defaults, >>>>>> because once you've checked out the demo app and put it into your >>>>>> local svn, you aren't going to get any future changes to it; but >>>>>> there might be a reason the defaults are what they are currently). >>>>>> >>>>>> I am more than happy to make these changes, submit patches, and >>>>>> write documentation. Just need guidance on what way you guys >>>>>> prefer >>>>>> it to go, so I don't spend time doing something you don't like and >>>>>> don't accept. >>>>>> >>>>>> Jonathan >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Blacklight-development mailing list >>>>> Blacklight-development at rubyforge.org >>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From rochkind at jhu.edu Wed Jun 24 12:40:25 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Wed, 24 Jun 2009 12:40:25 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <4A425608.6090403@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> <4A40F6B7.6050902@jhu.edu> <04413ADA-50F3-4C9E-956F-29A6FD7704FE@dang.com> <4A4102AA.7060503@jhu.edu> <4A410AEE.3010605@jhu.edu> <8B11FE0B-4710-4188-A0C9-3772C632C816@dang.com> <4A425608.6090403@jhu.edu> Message-ID: <4A4256F9.90206@jhu.edu> And, weird, if I delete the solr 'data' directory and start over from scratch... it works again. Sigh. Will investigate further. Jonathan Rochkind wrote: > Hey Jamie, so this is weird, I did get 'rake solr:marc:index' to work > once when I tried 'cd'ing into the bl plugin directory first, as you > advise. (Still thinking about/investigating how to fix it so this isn't > necessary). > > But when I try it a second time, with other records (or with the same > mrc record file), it doesn't work. With the below exception. Any ideas > at all? I don't know why it would work once, but not twice. I don't > think I've made any changes? Or do I need to go ask on the solrmarc > list about this? > > [rochkind at xs001 blacklight]$ rake solr:marc:index > MARC_RECORDS_PATH=../../../../data/lc_records.utf8.mrc > (in /opt/blacklight/bl-demo/rails/vendor/plugins/blacklight) > INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. > Error: Problem instantiating SolrCore > ERROR [main] (SolrCoreLoader.java:157) - Error: Problem instantiating > SolrCore > java.lang.reflect.InvocationTargetException > > > > > Jamie Orchard-Hays wrote: > >> OK. I just downloaded 2.2, untarred and tried out rake solr:marc:index. >> >> If I'm in the rails dir, it fails due to errors (the path is wrong). >> >> If I'm in the blacklight plugin dir, it works, because the path is >> right. >> >> The way to fix it is to have it use the __FILE__ location to orient >> the path properly. >> >> Except, that IS what the rake task does. So somehow when rake loads up >> all the tasks, it's not referring to the actual __FILE__ the task is >> created in. This is not the behavior I expect from Ruby. My >> understanding is that it should use that particular file is the >> reference. >> >> I'm perplexed. >> >> >> On Jun 23, 2009, at 1:03 PM, Jonathan Rochkind wrote: >> >> >> >>> I'd really like to help get things working so a new user can just >>> use the rake task included there out-of-the-box for a simple demo. >>> >>> Like a new user _can_ use "rake app:index:marc" right now, with the >>> newly checked out demo app, which is what the current instructions >>> say to use. But apparently that's not actually using SolrMarc, and >>> people think that task should be removed? >>> >>> In which case, the instructions which right now suggest that after >>> installing the demo app you should run "rake app:index:marc FILE=../ >>> data/lc_records.utf8.mrc", should be changed to suggest you run rake >>> solr:marc:index instead, right? I really think it's good to have >>> this available (and am happy to help make it so). >>> >>> But right now, if you try running sorl:marc:index, you get errors. >>> I'd really like to help fix things so this isn't so, so the >>> instructions can say "And kick off the rake task to index some test >>> data: rake solr:marc:index MARC_RECORDS_PATH=../data/ >>> test_data.utf8.mrc". >>> >>> That would be good, right? Is it possible for me to help make it >>> so? That's a much easier learning curve (for me and those that come >>> after me) to be able to start like this, then to say "Now, to index >>> some data, go figure out how to install and use SolrMarc." >>> >>> Jonathan >>> >>> Naomi Dushay wrote: >>> >>> >>>> Hi Jonathan, >>>> >>>> Oy, this is less obvious that we would have hoped. More >>>> documentation >>>> is definitely indicated. >>>> >>>> The demo index is basically intended as >>>> >>>> 1. a way for the plugin to have standalone tests, some of which need >>>> a SOLR index to run. >>>> 2. an exemplar. >>>> >>>> It's the 2. that's sub-optimal. >>>> >>>> We use solrmarc completely separately from the blacklight code to >>>> generate our index. Then we set up rails/config/solr.yml to point >>>> to >>>> the SOLR instance with the index we created. >>>> >>>> To do anything non-trivial with solrmarc probably implies you want to >>>> create a separate build process from solrmarc. You can certainly >>>> base >>>> it on the demo app ... but that's a very simple index that doesn't >>>> exploit much of the MARC data richness. >>>> >>>> I have it in my queue to flesh out the demo index more, so it's a >>>> better exemplar for MARC data. But I haven't gotten to it yet. And >>>> your notes make it clear that we have to cope with BOTH intentions >>>> above ... perhaps separately, or at least clearly documented. >>>> >>>> My apologies for this being so ... messy. Your experience, while >>>> frustrating for you, is very helpful to the project. >>>> >>>> And ... my recommendation is that you may want to use solrmarc >>>> independently, as UVa and Stanford do. You might start with the >>>> simple SOLR index fields the demo app uses, but populated from your >>>> data. However, you are likely to add a lot more fields to your >>>> index. >>>> >>>> You can check out http://searchworks-test.stanford.edu to see what >>>> our current blacklight efforts look like. You'll immediately notice >>>> how far we've strayed from the demo index. And if you saw the >>>> complexity of our search boosting formulae ... oy! >>>> >>>> - Naomi >>>> >>>> On Jun 23, 2009, at 9:28 AM, Jonathan Rochkind wrote: >>>> >>>> >>>> >>>> >>>>> Huh. But the default locations in there are NOT the (current) demo >>>>> app locations. So it sounds like they were intended to be, and I >>>>> can change them to be and submit a patch? >>>>> >>>>> I was going to ask Jamie if he wanted to have a chat in #blacklight >>>>> on the right way to get the rake task working (in a trivial case) >>>>> with the out-of-the-box demo app, but if he's not working on it >>>>> right now... >>>>> >>>>> I'll just do my best that seems to make sense to me, and submit a >>>>> patch? Jamie, there's no particular reason for the defaults chosen >>>>> except that they are in fact supposed to match the demo app? I'll >>>>> make it so. >>>>> >>>>> Jamie Orchard-Hays wrote: >>>>> >>>>> >>>>> >>>>>> Sigh. You're right. I did that because I wanted some default in >>>>>> there >>>>>> and used the demo app locations. You all may need to revisit it. >>>>>> Unfortunately, I'm not actively on this project right now. >>>>>> >>>>>> Jamie >>>>>> >>>>>> On Jun 23, 2009, at 11:37 AM, Jonathan Rochkind wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Heh, but the defaults for solr:marc:index are in general not >>>>>>> what's >>>>>>> in the demo app. >>>>>>> >>>>>>> rake solr:marc:index MARC_RECORDS_PATH=../data/lc_records.utf8.mrc >>>>>>> INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. >>>>>>> Error: Supplied Solr home directory does not exist: ../../../../ >>>>>>> jetty/solr >>>>>>> >>>>>>> I'm actually not sure what arg to give solr:marc:index to fix >>>>>>> where >>>>>>> it's looking for Solr home directory. But I can poke around in >>>>>>> the >>>>>>> code and figure it out. >>>>>>> >>>>>>> It seems to me that the defaults for solr:marc:index should match >>>>>>> what you get in the demo app though, yes? This can be changed by >>>>>>> changing the defaults, OR by changing the setup of the 'demo' app. >>>>>>> Which would you prefer? (I'd lean toward changing the defaults, >>>>>>> because once you've checked out the demo app and put it into your >>>>>>> local svn, you aren't going to get any future changes to it; but >>>>>>> there might be a reason the defaults are what they are currently). >>>>>>> >>>>>>> I am more than happy to make these changes, submit patches, and >>>>>>> write documentation. Just need guidance on what way you guys >>>>>>> prefer >>>>>>> it to go, so I don't spend time doing something you don't like and >>>>>>> don't accept. >>>>>>> >>>>>>> Jonathan >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Blacklight-development mailing list >>>>>> Blacklight-development at rubyforge.org >>>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>>> >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Blacklight-development mailing list >>>>> Blacklight-development at rubyforge.org >>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From jamie at dang.com Wed Jun 24 13:05:29 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Wed, 24 Jun 2009 13:05:29 -0400 Subject: [Blacklight-development] changing solrmarc indexing in demo app? In-Reply-To: <4A4256F9.90206@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A4003B8.6030102@jhu.edu> <4A40E2AC.3040604@jhu.edu> <4A40F508.30400@jhu.edu> <4A40F6B7.6050902@jhu.edu> <04413ADA-50F3-4C9E-956F-29A6FD7704FE@dang.com> <4A4102AA.7060503@jhu.edu> <4A410AEE.3010605@jhu.edu> <8B11FE0B-4710-4188-A0C9-3772C632C816@dang.com> <4A425608.6090403@jhu.edu> <4A4256F9.90206@jhu.edu> Message-ID: <2C3009AF-6CEA-4DCF-90DC-2E6D0BDAA799@dang.com> Dunno what it might be. On Jun 24, 2009, at 12:40 PM, Jonathan Rochkind wrote: > And, weird, if I delete the solr 'data' directory and start over > from scratch... it works again. Sigh. Will investigate further. > > Jonathan Rochkind wrote: >> Hey Jamie, so this is weird, I did get 'rake solr:marc:index' to work >> once when I tried 'cd'ing into the bl plugin directory first, as you >> advise. (Still thinking about/investigating how to fix it so this >> isn't >> necessary). >> >> But when I try it a second time, with other records (or with the same >> mrc record file), it doesn't work. With the below exception. Any >> ideas >> at all? I don't know why it would work once, but not twice. I don't >> think I've made any changes? Or do I need to go ask on the solrmarc >> list about this? >> >> [rochkind at xs001 blacklight]$ rake solr:marc:index >> MARC_RECORDS_PATH=../../../../data/lc_records.utf8.mrc >> (in /opt/blacklight/bl-demo/rails/vendor/plugins/blacklight) >> INFO [main] (MarcImporter.java:516) - Starting SolrMarc indexing. >> Error: Problem instantiating SolrCore >> ERROR [main] (SolrCoreLoader.java:157) - Error: Problem instantiating >> SolrCore >> java.lang.reflect.InvocationTargetException >> >> >> >> >> Jamie Orchard-Hays wrote: >> >>> OK. I just downloaded 2.2, untarred and tried out rake >>> solr:marc:index. >>> >>> If I'm in the rails dir, it fails due to errors (the path is wrong). >>> >>> If I'm in the blacklight plugin dir, it works, because the path is >>> right. >>> >>> The way to fix it is to have it use the __FILE__ location to orient >>> the path properly. >>> >>> Except, that IS what the rake task does. So somehow when rake >>> loads up >>> all the tasks, it's not referring to the actual __FILE__ the task is >>> created in. This is not the behavior I expect from Ruby. My >>> understanding is that it should use that particular file is the >>> reference. >>> >>> I'm perplexed. >>> >>> >>> On Jun 23, 2009, at 1:03 PM, Jonathan Rochkind wrote: >>> >>> >>> >>>> I'd really like to help get things working so a new user can just >>>> use the rake task included there out-of-the-box for a simple demo. >>>> >>>> Like a new user _can_ use "rake app:index:marc" right now, with the >>>> newly checked out demo app, which is what the current instructions >>>> say to use. But apparently that's not actually using SolrMarc, and >>>> people think that task should be removed? >>>> >>>> In which case, the instructions which right now suggest that after >>>> installing the demo app you should run "rake app:index:marc >>>> FILE=../ >>>> data/lc_records.utf8.mrc", should be changed to suggest you run >>>> rake >>>> solr:marc:index instead, right? I really think it's good to have >>>> this available (and am happy to help make it so). >>>> >>>> But right now, if you try running sorl:marc:index, you get errors. >>>> I'd really like to help fix things so this isn't so, so the >>>> instructions can say "And kick off the rake task to index some test >>>> data: rake solr:marc:index MARC_RECORDS_PATH=../data/ >>>> test_data.utf8.mrc". >>>> >>>> That would be good, right? Is it possible for me to help make it >>>> so? That's a much easier learning curve (for me and those that >>>> come >>>> after me) to be able to start like this, then to say "Now, to index >>>> some data, go figure out how to install and use SolrMarc." >>>> >>>> Jonathan >>>> >>>> Naomi Dushay wrote: >>>> >>>> >>>>> Hi Jonathan, >>>>> >>>>> Oy, this is less obvious that we would have hoped. More >>>>> documentation >>>>> is definitely indicated. >>>>> >>>>> The demo index is basically intended as >>>>> >>>>> 1. a way for the plugin to have standalone tests, some of which >>>>> need >>>>> a SOLR index to run. >>>>> 2. an exemplar. >>>>> >>>>> It's the 2. that's sub-optimal. >>>>> >>>>> We use solrmarc completely separately from the blacklight code to >>>>> generate our index. Then we set up rails/config/solr.yml to >>>>> point >>>>> to >>>>> the SOLR instance with the index we created. >>>>> >>>>> To do anything non-trivial with solrmarc probably implies you >>>>> want to >>>>> create a separate build process from solrmarc. You can certainly >>>>> base >>>>> it on the demo app ... but that's a very simple index that doesn't >>>>> exploit much of the MARC data richness. >>>>> >>>>> I have it in my queue to flesh out the demo index more, so it's a >>>>> better exemplar for MARC data. But I haven't gotten to it yet. >>>>> And >>>>> your notes make it clear that we have to cope with BOTH intentions >>>>> above ... perhaps separately, or at least clearly documented. >>>>> >>>>> My apologies for this being so ... messy. Your experience, while >>>>> frustrating for you, is very helpful to the project. >>>>> >>>>> And ... my recommendation is that you may want to use solrmarc >>>>> independently, as UVa and Stanford do. You might start with the >>>>> simple SOLR index fields the demo app uses, but populated from >>>>> your >>>>> data. However, you are likely to add a lot more fields to your >>>>> index. >>>>> >>>>> You can check out http://searchworks-test.stanford.edu to see >>>>> what >>>>> our current blacklight efforts look like. You'll immediately >>>>> notice >>>>> how far we've strayed from the demo index. And if you saw the >>>>> complexity of our search boosting formulae ... oy! >>>>> >>>>> - Naomi >>>>> >>>>> On Jun 23, 2009, at 9:28 AM, Jonathan Rochkind wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> Huh. But the default locations in there are NOT the (current) >>>>>> demo >>>>>> app locations. So it sounds like they were intended to be, and I >>>>>> can change them to be and submit a patch? >>>>>> >>>>>> I was going to ask Jamie if he wanted to have a chat in >>>>>> #blacklight >>>>>> on the right way to get the rake task working (in a trivial case) >>>>>> with the out-of-the-box demo app, but if he's not working on it >>>>>> right now... >>>>>> >>>>>> I'll just do my best that seems to make sense to me, and submit a >>>>>> patch? Jamie, there's no particular reason for the defaults >>>>>> chosen >>>>>> except that they are in fact supposed to match the demo app? I'll >>>>>> make it so. >>>>>> >>>>>> Jamie Orchard-Hays wrote: >>>>>> >>>>>> >>>>>> >>>>>>> Sigh. You're right. I did that because I wanted some default in >>>>>>> there >>>>>>> and used the demo app locations. You all may need to revisit it. >>>>>>> Unfortunately, I'm not actively on this project right now. >>>>>>> >>>>>>> Jamie >>>>>>> >>>>>>> On Jun 23, 2009, at 11:37 AM, Jonathan Rochkind wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Heh, but the defaults for solr:marc:index are in general not >>>>>>>> what's >>>>>>>> in the demo app. >>>>>>>> >>>>>>>> rake solr:marc:index MARC_RECORDS_PATH=../data/ >>>>>>>> lc_records.utf8.mrc >>>>>>>> INFO [main] (MarcImporter.java:516) - Starting SolrMarc >>>>>>>> indexing. >>>>>>>> Error: Supplied Solr home directory does not >>>>>>>> exist: ../../../../ >>>>>>>> jetty/solr >>>>>>>> >>>>>>>> I'm actually not sure what arg to give solr:marc:index to fix >>>>>>>> where >>>>>>>> it's looking for Solr home directory. But I can poke around in >>>>>>>> the >>>>>>>> code and figure it out. >>>>>>>> >>>>>>>> It seems to me that the defaults for solr:marc:index should >>>>>>>> match >>>>>>>> what you get in the demo app though, yes? This can be >>>>>>>> changed by >>>>>>>> changing the defaults, OR by changing the setup of the 'demo' >>>>>>>> app. >>>>>>>> Which would you prefer? (I'd lean toward changing the defaults, >>>>>>>> because once you've checked out the demo app and put it into >>>>>>>> your >>>>>>>> local svn, you aren't going to get any future changes to it; >>>>>>>> but >>>>>>>> there might be a reason the defaults are what they are >>>>>>>> currently). >>>>>>>> >>>>>>>> I am more than happy to make these changes, submit patches, and >>>>>>>> write documentation. Just need guidance on what way you guys >>>>>>>> prefer >>>>>>>> it to go, so I don't spend time doing something you don't >>>>>>>> like and >>>>>>>> don't accept. >>>>>>>> >>>>>>>> Jonathan >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Blacklight-development mailing list >>>>>>> Blacklight-development at rubyforge.org >>>>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Blacklight-development mailing list >>>>>> Blacklight-development at rubyforge.org >>>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Blacklight-development mailing list >>>>> Blacklight-development at rubyforge.org >>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Wed Jun 24 16:15:41 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Wed, 24 Jun 2009 16:15:41 -0400 Subject: [Blacklight-development] error in marc indexing? In-Reply-To: <4A3FFA39.3010403@virginia.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> Message-ID: <4A42896D.5020901@jhu.edu> Hey Robert, or anyone else. The original spec for the title value was actually: title_t = custom, removeTrailingPunct(245a) What does the 'custom' mean? I read (and re-read) what solrmarc docs there appear to be in the wiki, but still couldn't quite figure this out. If I try changing this to: title_t = custom, removeTrailingPunct(245a), first Then solrmarc complains. Should I instead change it to: title_t = removeTrailingPunct(245a), first Nope, that doesn't seem to work either, although it fails in an entirely different way. Do I need to give up the removeTrailingPunct in order to have the 'first'? Should I be asking this question on the solrmarc list instead? Thanks very much for any help. Jonathan Robert Haschart wrote: > Jonathan, > > Rather than changing the schema to not specify multiValued, you also > have a number of other options: > > change the spec for getting the title value from: > > title_t = 245a > > to : > > title_t = 245a, first > > which will get only the first occurance of a given field/subfield. > > or to: > > title_t = 245aa > > which will concatenate all 'a' subfields for a given 245 field and > return them as a single entry. > > or even: > > title_t = 245aa, first > > which will handle instances with multiple 'a' subfields in a single 245 > field as above, but still not die if multiple 245 fields are present. > > Lastly two features that were added to Solrmarc very recently (as in the > demo code does not yet have it) could help your situation. One is that > errors such as missing required field, duplicate, non-multiValued > fields, or field with unknown (to solr) names, will generate an error > message for that record, but allow the indexing to continue. > Second there is a new standard "custom" index function called > getSingleIndexEntry which could be used like this: > > title_t = custom, getSingleIndexEntry(245aa, true) > > It would process the 245aa field spec as above, and then if multiple > entries results it would select the longest result to use for the > title_t index entry, and if the second parameter to the function is > true (and if marc.include_errors is enabled) it will generate a > marc_error index entry containing the additional errorneous 245 field > data. > > -Bob Haschart > > > Jonathan Rochkind wrote: > > >> Cool. I'm not sure multiple 245$a's even technically _is_ "bad data". >> It's not neccesarily AACR2, but I found out the hard way recently on >> another project that we have LOTS of non-AACR2 data in our catalog, >> including AACR1 data, pre-AACR data, and data cataloged to rare books >> and manuscripts standards, none of which our catalogers actually >> consider 'bad'. >> So Ross suggests one answer, changing it to: >> >> title_t = custom, removeTrailingPunct(245a), first >> >> So it'll ignore the second 245. Another option might be figuring out >> how to set the solrmarc setup to concatenate two 245$a's, rather than >> ignore the second one, which would seem to me to be the actually >> appropriate thing to do in this case, barring letting title_t take >> multiple values. Is it possible to do that somehow? >> >> Anyone have an opinion on if either of these two things should be done >> in the standard out-of-the-box demo setup, to accomodate this kind of >> data? >> >> Jonathan >> >> >> >> Naomi Dushay wrote: >> >> >>> Jonathan, >>> >>> We have a TON of bad data. Lots of records with multiple 245a; lots >>> of records with other similar problems. >>> >>> We use solrmarc, and it nicely steps around these problems. There are >>> solrmarc lists; >>> >>> solrmarc-general at googlegroups.com >>> solrmarc-technical at googlegroups.com >>> >>> - Naomi >>> >>> >>> On Jun 22, 2009, at 1:44 PM, Jonathan Rochkind wrote: >>> >>> >>> >>> >>>> Trying to take one baby step at a time in getting a little demo of >>>> Blacklight with our demo, I'm trying to index a sample of our own >>>> local MARC records. >>>> >>>> I get an error from "rake app:index:marc", not sure exactly why, >>>> error below. Maybe errors in my MARC? There definitely _will_ be >>>> illegalities in my marc, would rather the indexer recovered from >>>> them somehow (or just skipped that record, if nothing else is >>>> possible), rather than punted the entire input. >>>> >>>> Should I switch to SolrMARC instead, is it a bit more forgiving? At >>>> one point I know I saw a pointer in the documentation to SolrMarc >>>> docs, to help you get started with solrmarc instead of the bundled >>>> ruby indexer... but now I can't seem to find it. >>>> >>>> Here's what "raek app:index:marc" is telling me (giving me an html >>>> error message even though it's a command-line rake task, which is a >>>> bit odd). >>>> rake aborted! >>>> >>>> >>>> >>>> Error 400 >>>> >>>>

HTTP ERROR: 400

ERROR: [53567] multiple values
>>>> encountered for non multiValued field title_t: [Honan's Handbook to
>>>> medical Europe, A ready reference book to the universities,
>>>> hospitals, clinics, laboratories and general medical work of the
>>>> principal cities of Europe]
>>>>

RequestURI=/solr/update

>>> href="http://jetty.mortbay.org/ >>>> ">Powered by Jetty://


>>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From ndushay at stanford.edu Wed Jun 24 16:21:09 2009 From: ndushay at stanford.edu (Naomi Dushay) Date: Wed, 24 Jun 2009 13:21:09 -0700 Subject: [Blacklight-development] relative paths in config.properties file? In-Reply-To: <4A42878E.2070201@jhu.edu> References: <4A425175.4010207@jhu.edu> <4A42878E.2070201@jhu.edu> Message-ID: <9233ADB1-74A2-42AD-939E-7A707A8C9594@stanford.edu> Great work, Jonathan. Thank you. - Naomi On Jun 24, 2009, at 1:07 PM, Jonathan Rochkind wrote: > > Okay, I think I figured this out. Indeed solrmarc interprets relative > paths in the config.properties as relative to the working directory. > (Which is kinda weird, and I still think it's probably broken, but > anyway). > > Since we want relative paths in there, Jamie already had a bit of > weird > workaround in the blacklight rake task for one value in particular, > solr.indexer.properties. But it didn't work around for solr.path. So > I've provided a more general purpose workaround, which is indeed > having > the rake task change working directory before calling solrmarc, so the > relative paths are predictable. > > > > Jonathan Rochkind wrote: >> I have a question about relative file paths in the config.properties >> file, for instance for properties like solr.path. >> >> Blacklight is attempting to distribute a sample/demo copy of >> Solrmarc, >> with some rake tasks to do indexing, to help people get started with >> some 'default' options better. To do this, it also distributes a >> config.properties file set to some out-of-the-box defaults. The >> config.properties file includes some relative file paths, in an >> attempt >> to make it "just work" with things as installed by the Blacklight >> demo >> app, regardless of where you've installed it. >> >> I'd think that relative file paths in the config.properties file >> would >> be relative to the config.properties file itself. But it appears that >> they may end up interpreted as relative to the _current working >> directory_. Which means whether your config.properties file actually >> works is dependent on what directory you are in when you try to >> start up >> solrmarc. Which is kind of confusing, and presents problems for >> Blacklight's attempt to provide a sample Solrmarc config which works >> automataically out of the box. >> >> Am I right about how these relative file paths are being interpreted? >> If so, is this desired/appropriate action, or is it a bug that >> should be >> fixed? (I'm afraid I'm not familiar with Java enough to prepare a >> patch >> myself). >> >> I suppose if I'm right, a workaround on the Blacklight end would be >> to >> have Blacklight do a 'cd' into the same directory as the config file >> before executing SolrMarc, so paths in there will indeed be >> interpreted >> relative to to the config.properties itself. But this adds some >> potential confusion. >> >> Thanks for any feedback! >> >> Jonathan >> >> > > --~--~---------~--~----~------------~-------~--~----~ > You received this message because you are subscribed to the Google > Groups "solrmarc-tech" group. > To post to this group, send email to solrmarc-tech at googlegroups.com > To unsubscribe from this group, send email to solrmarc-tech+unsubscribe at googlegroups.com > For more options, visit this group at http://groups.google.com/group/solrmarc-tech?hl=en > -~----------~----~----~----~------~----~------~--~--- > From rochkind at jhu.edu Wed Jun 24 16:24:39 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Wed, 24 Jun 2009 16:24:39 -0400 Subject: [Blacklight-development] relative paths in config.properties file? In-Reply-To: <9233ADB1-74A2-42AD-939E-7A707A8C9594@stanford.edu> References: <4A425175.4010207@jhu.edu> <4A42878E.2070201@jhu.edu> <9233ADB1-74A2-42AD-939E-7A707A8C9594@stanford.edu> Message-ID: <4A428B87.8000102@jhu.edu> I do have some questions about how I should submit this as a patch to Blacklight. Anyone want to help me through it in the IRC channel tomorrow? Naomi Dushay wrote: > Great work, Jonathan. Thank you. - Naomi > > On Jun 24, 2009, at 1:07 PM, Jonathan Rochkind wrote: > > >> Okay, I think I figured this out. Indeed solrmarc interprets relative >> paths in the config.properties as relative to the working directory. >> (Which is kinda weird, and I still think it's probably broken, but >> anyway). >> >> Since we want relative paths in there, Jamie already had a bit of >> weird >> workaround in the blacklight rake task for one value in particular, >> solr.indexer.properties. But it didn't work around for solr.path. So >> I've provided a more general purpose workaround, which is indeed >> having >> the rake task change working directory before calling solrmarc, so the >> relative paths are predictable. >> >> >> >> Jonathan Rochkind wrote: >> >>> I have a question about relative file paths in the config.properties >>> file, for instance for properties like solr.path. >>> >>> Blacklight is attempting to distribute a sample/demo copy of >>> Solrmarc, >>> with some rake tasks to do indexing, to help people get started with >>> some 'default' options better. To do this, it also distributes a >>> config.properties file set to some out-of-the-box defaults. The >>> config.properties file includes some relative file paths, in an >>> attempt >>> to make it "just work" with things as installed by the Blacklight >>> demo >>> app, regardless of where you've installed it. >>> >>> I'd think that relative file paths in the config.properties file >>> would >>> be relative to the config.properties file itself. But it appears that >>> they may end up interpreted as relative to the _current working >>> directory_. Which means whether your config.properties file actually >>> works is dependent on what directory you are in when you try to >>> start up >>> solrmarc. Which is kind of confusing, and presents problems for >>> Blacklight's attempt to provide a sample Solrmarc config which works >>> automataically out of the box. >>> >>> Am I right about how these relative file paths are being interpreted? >>> If so, is this desired/appropriate action, or is it a bug that >>> should be >>> fixed? (I'm afraid I'm not familiar with Java enough to prepare a >>> patch >>> myself). >>> >>> I suppose if I'm right, a workaround on the Blacklight end would be >>> to >>> have Blacklight do a 'cd' into the same directory as the config file >>> before executing SolrMarc, so paths in there will indeed be >>> interpreted >>> relative to to the config.properties itself. But this adds some >>> potential confusion. >>> >>> Thanks for any feedback! >>> >>> Jonathan >>> >>> >>> >> --~--~---------~--~----~------------~-------~--~----~ >> You received this message because you are subscribed to the Google >> Groups "solrmarc-tech" group. >> To post to this group, send email to solrmarc-tech at googlegroups.com >> To unsubscribe from this group, send email to solrmarc-tech+unsubscribe at googlegroups.com >> For more options, visit this group at http://groups.google.com/group/solrmarc-tech?hl=en >> -~----------~----~----~----~------~----~------~--~--- >> >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From eos8d at virginia.edu Wed Jun 24 16:25:35 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Wed, 24 Jun 2009 16:25:35 -0400 Subject: [Blacklight-development] error in marc indexing? In-Reply-To: <4A42896D.5020901@jhu.edu> References: <4A3FED46.10106@jhu.edu> <4A3FF59E.4030803@jhu.edu> <4A3FFA39.3010403@virginia.edu> <4A42896D.5020901@jhu.edu> Message-ID: <48060ED9-D2B0-4F30-BB2B-7CC1752A3CA1@virginia.edu> Hey, Jonathan. Bob will see your post on either list, but it's probably better to ask solrmarc specific questions on the solrmarc list, so we create an archive of the questions over there. There's a population of people who use solrmarc who do not use blacklight, and we want to take advantage of their expertise and leave documentation for them. Thanks! Bess On 24-Jun-09, at 4:15 PM, Jonathan Rochkind wrote: > Hey Robert, or anyone else. The original spec for the title value was > actually: > > title_t = custom, removeTrailingPunct(245a) > > What does the 'custom' mean? I read (and re-read) what solrmarc docs > there appear to be in the wiki, but still couldn't quite figure this > out. > > If I try changing this to: > > title_t = custom, removeTrailingPunct(245a), first > > Then solrmarc complains. Should I instead change it to: > > title_t = removeTrailingPunct(245a), first > > Nope, that doesn't seem to work either, although it fails in an > entirely > different way. > > Do I need to give up the removeTrailingPunct in order to have the > 'first'? > > Should I be asking this question on the solrmarc list instead? Thanks > very much for any help. > > Jonathan > > Robert Haschart wrote: >> Jonathan, >> >> Rather than changing the schema to not specify multiValued, you also >> have a number of other options: >> >> change the spec for getting the title value from: >> >> title_t = 245a >> >> to : >> >> title_t = 245a, first >> >> which will get only the first occurance of a given field/subfield. >> >> or to: >> >> title_t = 245aa >> >> which will concatenate all 'a' subfields for a given 245 field and >> return them as a single entry. >> >> or even: >> >> title_t = 245aa, first >> >> which will handle instances with multiple 'a' subfields in a single >> 245 >> field as above, but still not die if multiple 245 fields are present. >> >> Lastly two features that were added to Solrmarc very recently (as >> in the >> demo code does not yet have it) could help your situation. One is >> that >> errors such as missing required field, duplicate, non-multiValued >> fields, or field with unknown (to solr) names, will generate an error >> message for that record, but allow the indexing to continue. >> Second there is a new standard "custom" index function called >> getSingleIndexEntry which could be used like this: >> >> title_t = custom, getSingleIndexEntry(245aa, true) >> >> It would process the 245aa field spec as above, and then if multiple >> entries results it would select the longest result to use for the >> title_t index entry, and if the second parameter to the function is >> true (and if marc.include_errors is enabled) it will generate a >> marc_error index entry containing the additional errorneous 245 >> field >> data. >> >> -Bob Haschart >> >> >> Jonathan Rochkind wrote: >> >> >>> Cool. I'm not sure multiple 245$a's even technically _is_ "bad >>> data". >>> It's not neccesarily AACR2, but I found out the hard way recently on >>> another project that we have LOTS of non-AACR2 data in our catalog, >>> including AACR1 data, pre-AACR data, and data cataloged to rare >>> books >>> and manuscripts standards, none of which our catalogers actually >>> consider 'bad'. >>> So Ross suggests one answer, changing it to: >>> >>> title_t = custom, removeTrailingPunct(245a), first >>> >>> So it'll ignore the second 245. Another option might be figuring >>> out >>> how to set the solrmarc setup to concatenate two 245$a's, rather >>> than >>> ignore the second one, which would seem to me to be the actually >>> appropriate thing to do in this case, barring letting title_t take >>> multiple values. Is it possible to do that somehow? >>> >>> Anyone have an opinion on if either of these two things should be >>> done >>> in the standard out-of-the-box demo setup, to accomodate this kind >>> of >>> data? >>> >>> Jonathan >>> >>> >>> >>> Naomi Dushay wrote: >>> >>> >>>> Jonathan, >>>> >>>> We have a TON of bad data. Lots of records with multiple 245a; >>>> lots >>>> of records with other similar problems. >>>> >>>> We use solrmarc, and it nicely steps around these problems. >>>> There are >>>> solrmarc lists; >>>> >>>> solrmarc-general at googlegroups.com >>>> solrmarc-technical at googlegroups.com >>>> >>>> - Naomi >>>> >>>> >>>> On Jun 22, 2009, at 1:44 PM, Jonathan Rochkind wrote: >>>> >>>> >>>> >>>> >>>>> Trying to take one baby step at a time in getting a little demo of >>>>> Blacklight with our demo, I'm trying to index a sample of our own >>>>> local MARC records. >>>>> >>>>> I get an error from "rake app:index:marc", not sure exactly why, >>>>> error below. Maybe errors in my MARC? There definitely _will_ be >>>>> illegalities in my marc, would rather the indexer recovered from >>>>> them somehow (or just skipped that record, if nothing else is >>>>> possible), rather than punted the entire input. >>>>> >>>>> Should I switch to SolrMARC instead, is it a bit more forgiving? >>>>> At >>>>> one point I know I saw a pointer in the documentation to SolrMarc >>>>> docs, to help you get started with solrmarc instead of the bundled >>>>> ruby indexer... but now I can't seem to find it. >>>>> >>>>> Here's what "raek app:index:marc" is telling me (giving me an html >>>>> error message even though it's a command-line rake task, which >>>>> is a >>>>> bit odd). >>>>> rake aborted! >>>>> >>>>> >>>>> >>>>> Error 400 >>>>> >>>>>

HTTP ERROR: 400

ERROR: [53567] multiple values
>>>>> encountered for non multiValued field title_t: [Honan's Handbook  
>>>>> to
>>>>> medical Europe, A ready reference book to the universities,
>>>>> hospitals, clinics, laboratories and general medical work of the
>>>>> principal cities of Europe]
>>>>>

RequestURI=/solr/update

>>>> href="http://jetty.mortbay.org/ >>>>> ">Powered by Jetty://


>>>>> _______________________________________________ >>>>> Blacklight-development mailing list >>>>> Blacklight-development at rubyforge.org >>>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>>> Blacklightopac Blog http://blacklightopac.org/ >>>>> >>>>> >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Thu Jun 25 11:51:34 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 25 Jun 2009 11:51:34 -0400 Subject: [Blacklight-development] tests fail on pristine copy? Message-ID: <4A439D06.3080004@jhu.edu> The right way to run tests is to run 'rake spec' in the blacklight plugin directory, right? If I run that on an updated-to-trunk copy of the plugin from a 'demo' app checkout, I get 64 failures. Am I doing something wrong? Ah, it looks like most of the failures are 'couldn't connect to server'. For instance, couldn't connect to server /opt/blacklight/bl-demo/rails/vendor/plugins/blacklight/spec/../config/../vendor/gems/mwmitchell-rsolr-0.8.8/lib/rsolr/http_ client.rb:80:in `get' Okay. But I have the SOLR server running as configured by default with the 'demo' app. Do the rake tasks expect a solr server running on some other port than what's by default configured with the demo app? Or is this not the solr server it's trying to connect to, but something else? Please advise! Jonathan From jkeck at stanford.edu Thu Jun 25 12:17:53 2009 From: jkeck at stanford.edu (Jessie Keck) Date: Thu, 25 Jun 2009 09:17:53 -0700 Subject: [Blacklight-development] tests fail on pristine copy? In-Reply-To: <4A439D06.3080004@jhu.edu> References: <4A439D06.3080004@jhu.edu> Message-ID: <4A43A331.1040806@stanford.edu> Hi Jonathan, I believe the correct command would be rake solr:spec and rake solr:features -Jessie Jonathan Rochkind wrote: > The right way to run tests is to run 'rake spec' in the blacklight > plugin directory, right? > > If I run that on an updated-to-trunk copy of the plugin from a 'demo' > app checkout, I get 64 failures. > > Am I doing something wrong? > > Ah, it looks like most of the failures are 'couldn't connect to > server'. For instance, > couldn't connect to server > /opt/blacklight/bl-demo/rails/vendor/plugins/blacklight/spec/../config/../vendor/gems/mwmitchell-rsolr-0.8.8/lib/rsolr/http_ > > client.rb:80:in `get' > > > Okay. But I have the SOLR server running as configured by default with > the 'demo' app. Do the rake tasks expect a solr server running on > some other port than what's by default configured with the demo app? > Or is this not the solr server it's trying to connect to, but > something else? > > Please advise! > > Jonathan > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Thu Jun 25 12:30:28 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 25 Jun 2009 12:30:28 -0400 Subject: [Blacklight-development] tests fail on pristine copy? In-Reply-To: <4A43A331.1040806@stanford.edu> References: <4A439D06.3080004@jhu.edu> <4A43A331.1040806@stanford.edu> Message-ID: <4A43A624.5090603@jhu.edu> Yep, Bess helped me out with that in IRC too. There is one test that's failing on pristine copy. Bess suggests you guys may have forgotten to remove a task that's no longer neccesary? Either that, or bcrypt needs to be a gem dependency still. Scenario: Failed Login # features/user_account.feature:44 Given user with login "user1" and email "user1 at foo.com" and password "password" # features/step_definitions/user_steps.rb:1 And I am on the login page # features/step_definitions/webrat_steps.rb:6 When I fill in "Login" with "user1" # features/step_definitions/webrat_steps.rb:22 When I fill in "Password" with "wrong-password" # features/step_definitions/webrat_steps.rb:22 And I press "Login" # features/step_definitions/webrat_steps.rb:14 uninitialized constant BCrypt (NameError) /opt/blacklight/bl-demo/rails/vendor/plugins/blacklight/app/controllers/user_sessions_controller.rb:29:in `create' (eval):2:in `click_button' ./features/step_definitions/webrat_steps.rb:15:in `/^I press "([^\"]*)"$/' features/user_account.feature:49:in `And I press "Login"' Jessie Keck wrote: > Hi Jonathan, > I believe the correct command would be rake solr:spec and rake solr:features > > -Jessie > > Jonathan Rochkind wrote: > >> The right way to run tests is to run 'rake spec' in the blacklight >> plugin directory, right? >> >> If I run that on an updated-to-trunk copy of the plugin from a 'demo' >> app checkout, I get 64 failures. >> >> Am I doing something wrong? >> >> Ah, it looks like most of the failures are 'couldn't connect to >> server'. For instance, >> couldn't connect to server >> /opt/blacklight/bl-demo/rails/vendor/plugins/blacklight/spec/../config/../vendor/gems/mwmitchell-rsolr-0.8.8/lib/rsolr/http_ >> >> client.rb:80:in `get' >> >> >> Okay. But I have the SOLR server running as configured by default with >> the 'demo' app. Do the rake tasks expect a solr server running on >> some other port than what's by default configured with the demo app? >> Or is this not the solr server it's trying to connect to, but >> something else? >> >> Please advise! >> >> Jonathan >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From rochkind at jhu.edu Thu Jun 25 13:53:08 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 25 Jun 2009 13:53:08 -0400 Subject: [Blacklight-development] solr:marc:index rake task Message-ID: <4A43B984.2040901@jhu.edu> Finalizing up my fixes to the solr:marc:index rake task, which Bess is then going to help me with the patch to trunk. I'm confident this does what Jamie intended originally, and will help the starter user get started, as well as provide a nice easy path to taking baby steps to more customization. One question though: The rake task will (now) look in a standard place in your app for the solrmarc config.properties -- should this allow for you to have a different config.properties for development vs. production (or any other environment). Is that something people generally need to do, use different config.properties for dev vs production? I guess you would have different solr's located at different places for this, so, probably? From rochkind at jhu.edu Thu Jun 25 15:26:48 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 25 Jun 2009 15:26:48 -0400 Subject: [Blacklight-development] moving on: cite this, refworks, holdings Message-ID: <4A43CF78.5090805@jhu.edu> So, one baby step at a time. I now have the default 'demo' blacklight app up, and can index both the demo the records that come with blacklight, and/or a sample of 1000 records we've assembled from our own catalog. Now of course I have some more questions. The demo app has "Cite This" and "Export to Refworks" links for all records. Great. Except... they don't actually work. "Cite this" ends up only including the title of a record, and not it's author, date, etc., even if the record has those fields. "Export to Refworks" has Refworks giving me an error: "The direct export from Blacklight is not working at this time. We regret any inconvenience that this may cause." This seems to be true of the LC record set, of our JHU record set, or of the smaller blacklight.utf8 sample set -- the records used don't matter. Any idea what's up? Or should I just dive into debugging myself, and get acquainted with the code? If so, anyone have any pointers to help me figure out what files I want to be looking at, instead of going in blind? Secondly, I know that at least UVa has a Blacklight up that shows item holdings details on the record detail page. The demo blacklight doesn't seem to do that, even when I index a JHU marc sample that does have some holdings info in the marc. Bess or anyone else at UVa, can you help me understand how you've done that, with what parts of blacklight core, what parts of stuff in your custom app, etc? Jonathan From rochkind at jhu.edu Tue Jun 30 12:03:43 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 30 Jun 2009 12:03:43 -0400 Subject: [Blacklight-development] more stored marc issues Message-ID: <4A4A375F.8030805@jhu.edu> Okay, so now that I know what to look for, it looks to me like my app already SHOULD have been storing the marc. SolrMarc index.properties says: marc_display = FullRecordAsMARC My blacklight_config.rb says: SolrDocument.marc_source_field = :marc_display SolrDocument.marc_format_type = :marcxml So that should be right, right? Aha, wait, no! The config is storing as marc binary, the blacklight_config is saying it's xml. So, okay I'll try fixing that, change marc_format_type to... what should I change it to, what's the other value possible here, other than :marcxml? Can I find that documented somewhere if I look in the right place? Assuming that solves it.... This was all as checked out from demo app, so I should probably submit a patch here to make demo app internally consistent, yes? Which is kind of confusing, because this would be a patch to files I've already given Bess changes to, but haven't been applied yet. Patch to a patch! Oh well. Which is the 'preferred' marc storage type, if I have a choice, to make the demo app do out of the box? xml or binary? From rochkind at jhu.edu Tue Jun 30 12:06:18 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 30 Jun 2009 12:06:18 -0400 Subject: [Blacklight-development] more stored marc issues In-Reply-To: <4A4A375F.8030805@jhu.edu> References: <4A4A375F.8030805@jhu.edu> Message-ID: <4A4A37FA.8040501@jhu.edu> And wait, looking through blacklight_config.rb, I see both: SolrDocument.marc_source_field = :marc_display SolrDocument.marc_format_type = :marcxml And config[:raw_storage_type] = "marcxml" # name of solr field containing raw data config[:raw_storage_field] = "marc_display" Why does it look like there are two configs doing the same thing? Do they both need to be changed? Do they both need to be kept consistent? Jonathan Rochkind wrote: > Okay, so now that I know what to look for, it looks to me like my app > already SHOULD have been storing the marc. > > SolrMarc index.properties says: > > marc_display = FullRecordAsMARC > > > My blacklight_config.rb says: > > SolrDocument.marc_source_field = :marc_display > SolrDocument.marc_format_type = :marcxml > > So that should be right, right? Aha, wait, no! The config is storing as > marc binary, the blacklight_config is saying it's xml. So, okay I'll try > fixing that, change marc_format_type to... what should I change it to, > what's the other value possible here, other than :marcxml? Can I find > that documented somewhere if I look in the right place? > > Assuming that solves it.... > > This was all as checked out from demo app, so I should probably submit a > patch here to make demo app internally consistent, yes? Which is kind > of confusing, because this would be a patch to files I've already given > Bess changes to, but haven't been applied yet. Patch to a patch! Oh well. > > Which is the 'preferred' marc storage type, if I have a choice, to make > the demo app do out of the box? xml or binary? > > > > From jkeck at stanford.edu Tue Jun 30 12:19:22 2009 From: jkeck at stanford.edu (Jessie Keck) Date: Tue, 30 Jun 2009 09:19:22 -0700 Subject: [Blacklight-development] more stored marc issues In-Reply-To: <4A4A37FA.8040501@jhu.edu> References: <4A4A375F.8030805@jhu.edu> <4A4A37FA.8040501@jhu.edu> Message-ID: <4A4A3B0A.9040604@stanford.edu> You need to change :marcxml and "marcxml" to :marc21 and "marc21" if you're storing binary data. As far as which is better (binary or XML), well that's probably a bigger discussion that blacklight. However; Stanford is storing binary Marc21 and Virginia stores Marc-XML. I know that Virginia had expressed concern that sometimes the binary data could possibly be corrupted or not maintain the fields in the same order, but we have not run into that problem (with a set of records over 5.5 mill.) Also, we have rSpec tests testing the output of both Marc21 and Marc-XML and the tests show that the output is the same. -Jessie Keck Jonathan Rochkind wrote: > And wait, looking through blacklight_config.rb, I see both: > > SolrDocument.marc_source_field = :marc_display > SolrDocument.marc_format_type = :marcxml > > > And > > config[:raw_storage_type] = "marcxml" > # name of solr field containing raw data > config[:raw_storage_field] = "marc_display" > > > Why does it look like there are two configs doing the same thing? Do > they both need to be changed? Do they both need to be kept consistent? > > > Jonathan Rochkind wrote: >> Okay, so now that I know what to look for, it looks to me like my app >> already SHOULD have been storing the marc. >> >> SolrMarc index.properties says: >> >> marc_display = FullRecordAsMARC >> >> >> My blacklight_config.rb says: >> >> SolrDocument.marc_source_field = :marc_display >> SolrDocument.marc_format_type = :marcxml >> >> So that should be right, right? Aha, wait, no! The config is storing as >> marc binary, the blacklight_config is saying it's xml. So, okay I'll try >> fixing that, change marc_format_type to... what should I change it to, >> what's the other value possible here, other than :marcxml? Can I find >> that documented somewhere if I look in the right place? >> >> Assuming that solves it.... >> >> This was all as checked out from demo app, so I should probably submit a >> patch here to make demo app internally consistent, yes? Which is kind >> of confusing, because this would be a patch to files I've already given >> Bess changes to, but haven't been applied yet. Patch to a patch! Oh >> well. >> >> Which is the 'preferred' marc storage type, if I have a choice, to make >> the demo app do out of the box? xml or binary? >> >> >> >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Tue Jun 30 13:31:14 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 30 Jun 2009 13:31:14 -0400 Subject: [Blacklight-development] success! Thanks. Message-ID: <4A4A4BE2.40401@jhu.edu> So I have blacklight properly showing 'cite this' and 'refworks', now that I've made solrmarc and blacklight agree on whether the marc is being stored as XML or Marc21. I will file a patch to fix the demo app to get this consistent. I guess consistent to MarcXML, for now.