From erikhatcher at mac.com Wed Apr 1 00:10:12 2009 From: erikhatcher at mac.com (Erik Hatcher) Date: Wed, 01 Apr 2009 00:10:12 -0400 Subject: [Blacklight-development] on the SolrDocument In-Reply-To: References: <49D2B69A.2010303@stanford.edu> Message-ID: On Mar 31, 2009, at 10:08 PM, Jamie Orchard-Hays wrote: > Matt started a thread about this last week called "Blacklight.config > - thoughts". He put a gist up with his suggestion: > > http://gist.github.com/86244 > > Is that the direction you're thinking, Erik? Not quite. Matt's last message actually is along the lines of what I'm thinking. More on this on other replies... Erik From erikhatcher at mac.com Wed Apr 1 00:21:24 2009 From: erikhatcher at mac.com (Erik Hatcher) Date: Wed, 01 Apr 2009 00:21:24 -0400 Subject: [Blacklight-development] on the SolrDocument In-Reply-To: <5836B658-2589-4C50-8DE5-BB3DF2C1CA35@stanford.edu> References: <49D2B69A.2010303@stanford.edu> <5836B658-2589-4C50-8DE5-BB3DF2C1CA35@stanford.edu> Message-ID: <97CFBC79-2E23-48D2-A3B7-CF3F7F671448@mac.com> On Mar 31, 2009, at 10:34 PM, Naomi Dushay wrote: > I wish I understood this better. What you're saying about solr.yml > makes some sense to me - it's becoming a real hodge-podge of > information meant as a sort of abstraction of some basic concepts, > and it keeps growing. Basically what I'm saying is to *model* this stuff as a true _object model_, not config files. Let the customizations and such be tweaks to a real object model, not hideous looking fat YAML files. > This all started because SOLR field names were hardcoded into a > variety of places: views, helpers, blah blah. That seemed like a > bad idea - silly to constrain blacklight users to have specific > field names in their SOLR index to match specific blacklight > functionality. Definitely an abstraction is needed, no question. I dig that. > Maybe I could understand your suggestion better if you could explain > it a little more, in the context of one or more of our specific needs. Sure, just some thoughts thrown out here.... > > > Here is the type of information we're trying to make easily > configurable: > > individual item view: > html title HTML title sounds like an override of a template, or partial. But could you elaborate on what difference in item title UVa wants and Stanford wants? > heading (most prominent text pertinent to individual item on the > page) Again, a template override. > what data to display Again, a template override. This likely means that the detail template should be broken into several smaller aggregated partial templates that can be overridden leaner and cleaner. Although this does perhaps lend itself to having the fields listed in a config file. But there could be many different types of items that need different configurations - could get messy quickly. Doing this at the template level and working hard on having as much common as possible seems the first steps though. > search results view: > heading data for each item > what data will be displayed Again, template overrides for these. > facets: > how to label them > how to order them Trickier business for these. But this could be done in solrconfig.xml - labels come from &facet.field={!key=YourCustomLabel}subject_facet, and of course ordering them can be specified there as well. There's other things to consider here... like is it better to have different search views (like UVa's general catalog vs. music) defined as separate request handlers in solrconfig.xml? Or should that configuration be more dynamically magically defined in Ruby? Or perhaps a little hybrid where there is an object model that defines these things and then a solrconfig.xml is generated. > Currently, we seem to be comfortable with the expectation that > blacklight will run on top of SOLR. Blacklight without Solr would be, well.... mighty dark :) > We know it won't always be marc. We know field names will vary. > We know different institutions will be writing code at the app level > to override lots of stuff: look & feel, various behaviors, whatever. Right on, and I totally dig that. I just don't think a .yml file is how you want the bulk of these overrides/customizations to occur. > How can we make it easy for blacklight adopters to get up and > running, yet maximize customizability with a minimum of burden (for > those installations that are okay with a fairly vanilla approach)? > We also want to make the plugin feature rich without shoving > requirements down adopters throats. I think you're on the right track... but rethinking how this config stuff works is the main point here. Erik From eos8d at virginia.edu Wed Apr 1 09:44:02 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Wed, 1 Apr 2009 09:44:02 -0400 Subject: [Blacklight-development] on the SolrDocument In-Reply-To: <49D2B69A.2010303@stanford.edu> References: <49D2B69A.2010303@stanford.edu> Message-ID: On Mar 31, 2009, at 8:34 PM, Jessie Keck wrote: > 1. it seems like the id field name and the marc field name are > hardcoded in there. > 2. it seems like the code is set up only for marcxml. Yes, this is true, but only because I wanted to solve one problem at a time. It worked, we got some working code out there that did task A, and now it's time to move onto task B. Let's go! > Naomi and I kicked around a few ideas. What do you think of this: > > a. adding storage_type as a property to solr.yml for "raw metadata > type". Currently might only be marc21 and marcxml, but Vernon is > interested in DC, and we'll be interested in MODS eventually ... Yes, this seems great to me. I'm not sure how this will fit with the rest of the conversation in this thread, but within the current code context this seems like a very good idea. > b. have the SolrDocument initialize method check that property and > initialize the class as appropriate. For example, marcxml uses the > marc xmlreader, but marc21 uses the marc reader. > DC might use something else, etc. Yes, again, no problem. So you're picturing two additional solr fields, yes? raw_storage - contains marc21 / marcxml / DC / MODS raw_storage_type - contains strings like "marc21" / "marcxml" / "DC" / "MODS" for switching on and knowing what is stored in raw_storage > I am currently working on this in line with my MLA/APA citation > work. What I have done is added a new field to the solr.yml file > and the DisplayFields module of storage_type > (next to the marc_storage_field). Hopefully in the future the > storage_type will be a solr field in the solr doc so that each > record can be handled according to its data type. > I am then doing a case statement on storage_type to instantiate the > MARC::Record object from either MARC21 or MARC-XML. Great. > Then in the SolrDocument class's initialize method I am setting > self.marc to that MARC::Record object so that the xml to marc_xml > methods don't need any changes. So maybe I'm having some java / ruby disconnect here, but it seems to me that what we actually want are inheriting classes, right? So if we initialize a SolrDocument (which, Erik is right, we should rename BlacklightDocument) and find out it has stored marc-xml, it in turn initializes a BlacklightMarcxmlDocument which has all the marc-xml specific stuff in it. Or it initializes a BlacklightDCDocument or a BlacklightMODSDocument. I like this approach for its separation of concerns, and maybe if we can identify a few core classes like this that we want to support it will be a concrete enough statement of the problem that I'll be able to get my head around more of what Erik and Matt are proposing. However, I'm a big believer in doing one thing at a time and committing frequently, so feel free to save the inheriting classes thing for a later iteration. > Also, I plan on extending the SolrDocument class out a little more > for my citations. I will have to_mla and to_apa methods of the > SolrDocument object so that the citations can be easily called. Yes! This is exactly why I wanted to get it done, so you'd have a place to put the citation work. However, the citation work will probably be object specific as well, right? That needn't change what you're writing right this minute, but for me it's another argument in favor of the subclasses. > I have working code written to do this. I will be extending the > current SolrDocument spec to test both flavors before I move on (and > I may need to ask for help with the spec). Awesome! I should be on chat most of the time for the next several days if you want to ask me any questions. And Jamie is a great teacher of RSpec. I spent most of last Friday with him working on this stuff and it was incredibly helpful. Bess From rochkind at jhu.edu Wed Apr 1 10:50:11 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Wed, 01 Apr 2009 10:50:11 -0400 Subject: [Blacklight-development] svn practices In-Reply-To: <5768FFD9-1F50-4CFC-9595-294B364C03D9@dang.com> References: <58A68FA9-D6D7-4E9E-A11F-169845C86080@stanford.edu> <5768FFD9-1F50-4CFC-9595-294B364C03D9@dang.com> Message-ID: <49D37F23.2080503@jhu.edu> I've found this text helpful in getting some basic grounding in how to use svn for release management: http://producingoss.com/en/development-cycle.html One thing that seems a bit odd to me is releasing a 2.0 release _before_ anyone is actually running it. With Xerxes, generally me and David run a release in 'beta' first, invariably find a few more bugs and fix them, and THEN tag a release for the public who will be even less tolerant of bugs than me and David. Jonathan Jamie Orchard-Hays wrote: > That's right. > > One thing I noticed is that branches/ now has 2.0/ with 2.0.0/ inside > of that. What we really want is something like 2.0-stable. Then, if we > need to make bug fixes to that branch, we make them in that branch and > then tag a release such as releases/blacklight-2.0.2 (it looks like > you're using releases/ rather than tags/ for this). > > So in a nutshell, trunk/ is where the main-line development is. > branches/ is where our stable and experimental branches go (they can > be committed to). releases/ is where we tag releases and we don't > commit changes (they should be considered read-only). > > Jamie > > > > On Mar 31, 2009, at 4:18 PM, Bess Sadler wrote: > > >> I think what we decided was (and correct me if I'm wrong here, >> folks) that active development that seems unlikely to step on toes >> goes in trunk. Most development will take place there in trunk, >> since we're all working on pretty different stuff. However, when you >> need to do something that's a pretty big departure, a big refactor, >> or a move to a different library, you want to create a branch for >> that until you get it working. You should not check anything into >> trunk that will break the build. >> >> Then, when we get close to another release, we branch it and tag it. >> >> Sound right? >> >> Bess >> >> On Mar 30, 2009, at 1:42 PM, Naomi Dushay wrote: >> >> >>> Er ... >>> >>> I thought we agreed that for an open source project, we were going to >>> do this: >>> >>> 1. have release tags >>> 2. do active development in branches >>> 3. put stable pre-release development in the trunk (for "nightly" >>> type builds) >>> >>> ?? >>> >>> So Matt's changes should go in the active development branch ... >>> wherever that is?? >>> >>> - Naomi >>> >>> >>> On Mar 28, 2009, at 4:37 PM, Bess Sadler wrote: >>> >>> >>>> Hey, folks. >>>> >>>> I had a pretty busy day: >>>> >>>> - I wrote (or rather, copied shamelessly from the Ruby Cookbook) a >>>> Rakefile to build release packages. >>>> - I then used that rakefile to create a 2.0.0 release and upload it >>>> to rubyforge.org >>>> - it has release notes from jira >>>> - I downloaded it and tested it and everything worked fine, but >>>> please other folks test it too >>>> - I re-organized our subversion tree so that everything that was >>>> previously in branches/rails-engines/trunk/ is now in /trunk and has >>>> been tagged under releases and branches. I also trimmed some un-used >>>> branches. >>>> - I generated the rdocs for the blacklight plugin and uploaded them >>>> to http://blacklight.rubyforge.org. These are currently quite sparse >>>> and I would love it if we could flesh them out and take advantage of >>>> rdocs for our next release. >>>> >>>> What else do we need to do to consider this 2.0 release done? If we >>>> can, I'd love to be able to make some announcements on Monday, so if >>>> anyone is available tomorrow would you please help me double-check >>>> these changes? >>>> >>>> Thank you! >>>> >>>> Bess >>>> _______________________________________________ >>>> Blacklight-development mailing list >>>> Blacklight-development at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/blacklight-development >>>> Blacklightopac Blog http://blacklightopac.org/ >>>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From rochkind at jhu.edu Wed Apr 1 10:56:37 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Wed, 01 Apr 2009 10:56:37 -0400 Subject: [Blacklight-development] on the SolrDocument In-Reply-To: References: <49D2B69A.2010303@stanford.edu> Message-ID: <49D380A5.9070403@jhu.edu> Bess Sadler wrote: > > raw_storage - contains marc21 / marcxml / DC / MODS > raw_storage_type - contains strings like "marc21" / "marcxml" / "DC" / > "MODS" for switching on and knowing what is stored in raw_storage > This makes sense to me. If possible, use an existing web vocabulary for those raw_storage_type strings, instead of using your own strings. I don't think Internet Content Type (MIME) is sufficient. But maybe there's a URI-based vocabulary that covers all those? Maybe something in DC? Jonathan > > >> I am currently working on this in line with my MLA/APA citation >> work. What I have done is added a new field to the solr.yml file >> and the DisplayFields module of storage_type >> (next to the marc_storage_field). Hopefully in the future the >> storage_type will be a solr field in the solr doc so that each >> record can be handled according to its data type. >> I am then doing a case statement on storage_type to instantiate the >> MARC::Record object from either MARC21 or MARC-XML. >> > > Great. > > >> Then in the SolrDocument class's initialize method I am setting >> self.marc to that MARC::Record object so that the xml to marc_xml >> methods don't need any changes. >> > > So maybe I'm having some java / ruby disconnect here, but it seems to > me that what we actually want are inheriting classes, right? So if we > initialize a SolrDocument (which, Erik is right, we should rename > BlacklightDocument) and find out it has stored marc-xml, it in turn > initializes a BlacklightMarcxmlDocument which has all the marc-xml > specific stuff in it. Or it initializes a BlacklightDCDocument or a > BlacklightMODSDocument. I like this approach for its separation of > concerns, and maybe if we can identify a few core classes like this > that we want to support it will be a concrete enough statement of the > problem that I'll be able to get my head around more of what Erik and > Matt are proposing. > > However, I'm a big believer in doing one thing at a time and > committing frequently, so feel free to save the inheriting classes > thing for a later iteration. > > >> Also, I plan on extending the SolrDocument class out a little more >> for my citations. I will have to_mla and to_apa methods of the >> SolrDocument object so that the citations can be easily called. >> > > Yes! This is exactly why I wanted to get it done, so you'd have a > place to put the citation work. However, the citation work will > probably be object specific as well, right? That needn't change what > you're writing right this minute, but for me it's another argument in > favor of the subclasses. > > >> I have working code written to do this. I will be extending the >> current SolrDocument spec to test both flavors before I move on (and >> I may need to ask for help with the spec). >> > > Awesome! I should be on chat most of the time for the next several > days if you want to ask me any questions. And Jamie is a great teacher > of RSpec. I spent most of last Friday with him working on this stuff > and it was incredibly helpful. > > Bess > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From rossfsinger at gmail.com Wed Apr 1 11:25:41 2009 From: rossfsinger at gmail.com (Ross Singer) Date: Wed, 1 Apr 2009 11:25:41 -0400 Subject: [Blacklight-development] on the SolrDocument In-Reply-To: <49D380A5.9070403@jhu.edu> References: <49D2B69A.2010303@stanford.edu> <49D380A5.9070403@jhu.edu> Message-ID: <23b83f160904010825p6702f123ka937215391419bb6@mail.gmail.com> Jangle has created a thesaurus for these sorts of specific format problems. http://jangle.org/vocab/formats I have been wanting to move it out the jangle.org/vocab space into something a little more generic, however - if there's a mutual interest in something like this, shall we move it to the metadata registry? Of course we could always try to register an info:uri :P -Ross. On Wed, Apr 1, 2009 at 10:56 AM, Jonathan Rochkind wrote: > Bess Sadler wrote: >> >> raw_storage - contains marc21 / marcxml / DC / MODS >> raw_storage_type - contains strings like "marc21" / "marcxml" / "DC" / >> ?"MODS" for switching on and knowing what is stored in raw_storage >> > > This makes sense to me. If possible, use an existing web vocabulary for > those raw_storage_type strings, instead of using your own strings. ?I don't > think Internet Content Type (MIME) is sufficient. But maybe there's a > URI-based vocabulary that covers all those? Maybe something in DC? > > Jonathan > > >> >> >>> >>> I am currently working on this in line with my MLA/APA citation ?work. >>> ?What I have done is added a new field to the solr.yml file ?and the >>> DisplayFields module of storage_type >>> (next to the marc_storage_field). ?Hopefully in the future the >>> ?storage_type will be a solr field in the solr doc so that each ?record can >>> be handled according to its data type. >>> I am then doing a case statement on storage_type to instantiate the >>> ?MARC::Record object from either MARC21 or MARC-XML. >>> >> >> Great. >> >> >>> >>> Then in the SolrDocument class's initialize method I am setting >>> ?self.marc to that MARC::Record object so that the xml to marc_xml ?methods >>> don't need any changes. >>> >> >> So maybe I'm having some java / ruby disconnect here, but it seems to ?me >> that what we actually want are inheriting classes, right? So if we >> ?initialize a SolrDocument (which, Erik is right, we should rename >> ?BlacklightDocument) and find out it has stored marc-xml, it in turn >> ?initializes a BlacklightMarcxmlDocument which has all the marc-xml >> ?specific stuff in it. Or it initializes a BlacklightDCDocument or a >> ?BlacklightMODSDocument. I like this approach for its separation of >> ?concerns, and maybe if we can identify a few core classes like this ?that >> we want to support it will be a concrete enough statement of the ?problem >> that I'll be able to get my head around more of what Erik and ?Matt are >> proposing. >> >> However, I'm a big believer in doing one thing at a time and ?committing >> frequently, so feel free to save the inheriting classes ?thing for a later >> iteration. >> >> >>> >>> Also, I plan on extending the SolrDocument class out a little more ?for >>> my citations. ?I will have to_mla and to_apa methods of the ?SolrDocument >>> object so that the citations can be easily called. >>> >> >> Yes! This is exactly why I wanted to get it done, so you'd have a ?place >> to put the citation work. However, the citation work will ?probably be >> object specific as well, right? That needn't change what ?you're writing >> right this minute, but for me it's another argument in ?favor of the >> subclasses. >> >> >>> >>> I have working code written to do this. ?I will be extending the ?current >>> SolrDocument spec to test both flavors before I move on (and ?I may need to >>> ask for help with the spec). >>> >> >> Awesome! I should be on chat most of the time for the next several ?days >> if you want to ask me any questions. And Jamie is a great teacher ?of RSpec. >> I spent most of last Friday with him working on this stuff ?and it was >> incredibly helpful. >> >> Bess >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From goodieboy at gmail.com Wed Apr 1 12:03:26 2009 From: goodieboy at gmail.com (Matt Mitchell) Date: Wed, 1 Apr 2009 12:03:26 -0400 Subject: [Blacklight-development] on the SolrDocument In-Reply-To: References: <49D2B69A.2010303@stanford.edu> Message-ID: We are talking about a few different things here I think. General app configuration, field mapping and domain issues. My thought for app configuration is that it could be a global, RAILS_ENV sensitive hash; Blacklight.config. The solr url and other configuration settings could be handled here using ruby. For display, we could chunk out as much as possible into partials. I think this is a good idea in general. Overriding files provides a high level approach to customizing and it's simple. Helper methods are another way, but can provide even more flexibility. Those are two things that Rails does out of the box and we should use them as much as possible without getting too complicated. So instead of: <%= DisplayFields.index_view[:show_link] %> Why not just: <%= show_link %> For the page title example that Naomi brought up, this could be a helper too: <%= doc_title %> The built-in, BL doc_title helper would have a little bit of logic if needed, and your app could override it too. And if done correctly, it could even override and call the built-in/super doc_title. Jamie, your question about just overriding a file in app/models... If you wanted to do it that way, you would create a new file at app/models/solr_document.rb. You would then have to call "require_dependency" to load the SolrDocument from the BL plugin. This would get you the original SolrDocument class definition. You'd then open up the class, and "include" another module that has the custom behavior. You can then call super on methods your module overrides. However, there seems to be a need for field mapping for the solr documents, stuff that's more in the model domain. I think putting this into the document model is the way to go for this, and not an external configuration mapping. My previous post might have sounded like I was saying BL should provide this functionality out-of-the-box; some kind of sub-framework for customizing solr-docs with "extend". That's not what I was suggesting though, that would definitely be over-engineering the plugin. I think that BL should limit the domain specific behavior if provides out of the box. And BL should provide an easy access point to when and where solr documents are created. Then specific institutions/implementations of BL could leverage this point to customize their models, using "extend" etc.. SolrHelper is the perfect spot for this. If we always use SolrHelper to create solr docs, then we can easily inject behavior into each document, and the rest of the app (partials, helpers) don't have to worry about it. Examples... There could be a BlacklightDisplayFields module that would be responsible for mapping generic BL fields like "title", "author". This module would intercept calls to those fields and map them to the correct solr field: module BlacklightDisplayFields def title self[:title_t] || 'Untitled' end def author self[:author_t] end end solr_doc.extend BlacklightDisplayFields <%= solr_doc.title %> Another module would be responsible for querying an availablilty service. So, you'd only inject this module into solr docs that need this functionality, like library catalog items. One module could be responsible for providing links to a Fedora repository. You'd only apply this module to solr docs that have a field of :fedora_pid, for example. The current SolrDocument marc stuff could be applied using a module as well. Because the current implementation is an OpenStruct though, we loose the original ability to access fields using a strings or symbols. For example, the standard solr doc coming from RSolr can be used like doc['title'] or doc[:title]. If we were to use extend instead, we'd still have that functionality. It could be done like this: module SolrMARCDocument def marc @marc ||= ( reader = MARC::XMLReader.new(StringIO.new(self.marc_display)).to_a reader[0] ) end def marc_xml self.marc.to_xml.to_s end def to_xml self.marc_xml end end Then: solr_doc.extend SolrMARCDocument solr_doc.marc_xml So there wouldn't really be a concrete SolrDocument class. There would be solr documents (hashes) with mixins applied to create richer objects. The code that handles the "extend" could be in SolrHelper. One method, that you could overridein your own app. ------------------ I guess what I'm saying is that we (could): - keep mapping/behaviors in the models (solr documents). We can do this in a super elegant way by using standard ruby techniques (extend, include). - keep the built-in BL behaviors as generic/simple as possible - have a single point in BL that has the opportunity to modify each solr doc that Blacklight is going to use (SolrHelper) Matt 2009/3/31 Jamie Orchard-Hays > I see value in this, but I'm wondering if it's getting too clever. Isn't > the idea of having Blacklight as a plugin so that end users can change the > default functionality/behavior by overriding the files in the > RAILS_HOME/apps dir? Is this over-engineering the plugin? > I'm not asking this rhetorically. I'm wondering. > > For example, if someone doesn't want our to_xml in SolrDocument, can't they > create or new SolrDocument in RAILS_HOME/apps/models/ and have at it? (I'm > unclear if the Rails Engine plugin allows changing models this way.) > > Jamie > > > > > On Mar 31, 2009, at 10:07 PM, Matt Mitchell wrote: > > I second the "good work"! > > For the config stuff, I've got some stuff I'm going to try and get in > sometime this week. Using a standard hash, and some basic switching based on > the RAILS_ENV value. > > Erik, I was thinking the same thing about the SolrDocument. What about solr > documents that don't use any marc? There may be lots of different data types > coming down the pipeline in the future. This is where inheritance will be a > problem. > > The thing we did in the UVa implementation was to attach behaviors to solr > documents, instead of wrapping them in another class. To do this, you can > use "extend" on an object. Sorry for the code examples, but it's easier for > me to explain... > > module MARCCommon > # some common marc stuff here? > end > > module MARC21Doc > include MARCCommon > def to_marc > > end > end > > module MARCXmlDoc > include MARCCommon > def to_xml > > end > end > > module DigitalLibraryDoc > def images > > end > end > > # in SolrHelper/controller, or some other custom "finder": > docs = solr.select(:q=>'*:*') > docs.each do |doc| > doc.extend MARCXmlDoc if doc[:marc_xml] > doc.extend MARC21Doc if doc[:marc_21] > doc.extend DigitalLibraryDoc if doc[:digital_lib] > end > > Later on in your view code (for example) > > @docs.each do |doc| > if doc.kind_of?(DigitalLibraryDoc) > puts 'I'm a digilib doc!' > end > end > > That way, each object can have it's own behavior, depending on its own > characteristics. > > Matt > > On Tue, Mar 31, 2009 at 8:34 PM, Jessie Keck wrote: > >> Hi all, >> >> Great job on getting release 2.0.0 out. Good work. We'll be upgrading >> in the next few days, I hope. >> >> So Naomi and I were looking at the SolrDocument class, and we realized a >> couple of things: >> >> 1. it seems like the id field name and the marc field name are hardcoded >> in there. >> 2. it seems like the code is set up only for marcxml. >> >> Naomi and I kicked around a few ideas. What do you think of this: >> >> a. adding storage_type as a property to solr.yml for "raw metadata type". >> Currently might only be marc21 and marcxml, but Vernon is interested in DC, >> and we'll be interested in MODS eventually ... >> >> b. have the SolrDocument initialize method check that property and >> initialize the class as appropriate. For example, marcxml uses the marc >> xmlreader, but marc21 uses the marc reader. DC might use something else, >> etc. >> >> >> I am currently working on this in line with my MLA/APA citation work. >> What I have done is added a new field to the solr.yml file and the >> DisplayFields module of storage_type (next to the marc_storage_field). >> Hopefully in the future the storage_type will be a solr field in the solr >> doc so that each record can be handled according to its data type. >> I am then doing a case statement on storage_type to instantiate the >> MARC::Record object from either MARC21 or MARC-XML. >> >> Then in the SolrDocument class's initialize method I am setting self.marc >> to that MARC::Record object so that the xml to marc_xml methods don't need >> any changes. >> >> Also, I plan on extending the SolrDocument class out a little more for my >> citations. I will have to_mla and to_apa methods of the SolrDocument object >> so that the citations can be easily called. >> >> I have working code written to do this. I will be extending the current >> SolrDocument spec to test both flavors before I move on (and I may need to >> ask for help with the spec). >> >> Thanks, >> -Jessie >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndushay at stanford.edu Wed Apr 1 13:16:33 2009 From: ndushay at stanford.edu (Naomi Dushay) Date: Wed, 1 Apr 2009 10:16:33 -0700 Subject: [Blacklight-development] on the SolrDocument In-Reply-To: References: <49D2B69A.2010303@stanford.edu> Message-ID: This is a great discussion. It would be even better if we could all be together in a room with a whiteboard, but oh well. Great job on the emails with enough specifics to make a reasonable substitute. I'm on board with keeping the plugin generic ... but if we do that, we want to have exemplars. It would be great if Jessie and I were the only RoR newbs in potential blacklight adopters ... but I suspect we're not. And maybe I'm being Chicken Little here, but the more straightforward the code is, the better, for where I am currently. That would favor Matt's SolrDoc approach vs. ostruct. Maybe I'll retract that statement in the near future, but let's adhere to KISS whenever we can. (Of course, the crux of this is Jamie's interpretation of "simple" vs. "mine" :-) ) These exemplars demonstrating how to work with Marc21, Marcxml, DC, etc. might be part of the demo app, or there might be a couple of exemplar apps. We know we've got traction with marc exemplar. From what Jessie was saying, I think we only need "marc"; the distinction between marc21 and marcxml is just Marc::XMLReader vs. MARC::Reader in the initializer, AFAIK. But this is a quibble. Once Vernon codes for DC, hopefully we can grab his stuff as an exemplar as well. I see the to-be-entered JIRA tasks for refactoring as: 1. how we initialize the code 2. replacing solr.yml with overridable code we also have 3. cleanly splitting behaviors between plugin and exemplars. Thoughts? - Naomi On Apr 1, 2009, at 9:03 AM, Matt Mitchell wrote: > We are talking about a few different things here I think. General > app configuration, field mapping and domain issues. My thought for > app configuration is that it could be a global, RAILS_ENV sensitive > hash; Blacklight.config. The solr url and other configuration > settings could be handled here using ruby. > > For display, we could chunk out as much as possible into partials. I > think this is a good idea in general. Overriding files provides a > high level approach to customizing and it's simple. Helper methods > are another way, but can provide even more flexibility. Those are > two things that Rails does out of the box and we should use them as > much as possible without getting too complicated. So instead of: > > <%= DisplayFields.index_view[:show_link] %> > > Why not just: > > <%= show_link %> > > For the page title example that Naomi brought up, this could be a > helper too: > > <%= doc_title %> > > The built-in, BL doc_title helper would have a little bit of logic > if needed, and your app could override it too. And if done > correctly, it could even override and call the built-in/super > doc_title. > > Jamie, your question about just overriding a file in app/models... > If you wanted to do it that way, you would create a new file at app/ > models/solr_document.rb. You would then have to call > "require_dependency" to load the SolrDocument from the BL plugin. > This would get you the original SolrDocument class definition. You'd > then open up the class, and "include" another module that has the > custom behavior. You can then call super on methods your module > overrides. > > However, there seems to be a need for field mapping for the solr > documents, stuff that's more in the model domain. I think putting > this into the document model is the way to go for this, and not an > external configuration mapping. My previous post might have sounded > like I was saying BL should provide this functionality out-of-the- > box; some kind of sub-framework for customizing solr-docs with > "extend". That's not what I was suggesting though, that would > definitely be over-engineering the plugin. > > I think that BL should limit the domain specific behavior if > provides out of the box. And BL should provide an easy access point > to when and where solr documents are created. Then specific > institutions/implementations of BL could leverage this point to > customize their models, using "extend" etc.. SolrHelper is the > perfect spot for this. If we always use SolrHelper to create solr > docs, then we can easily inject behavior into each document, and the > rest of the app (partials, helpers) don't have to worry about it. > > Examples... > > There could be a BlacklightDisplayFields module that would be > responsible for mapping generic BL fields like "title", "author". > This module would intercept calls to those fields and map them to > the correct solr field: > > module BlacklightDisplayFields > def title > self[:title_t] || 'Untitled' > end > def author > self[:author_t] > end > end > > solr_doc.extend BlacklightDisplayFields > > <%= solr_doc.title %> > > Another module would be responsible for querying an availablilty > service. So, you'd only inject this module into solr docs that need > this functionality, like library catalog items. > > One module could be responsible for providing links to a Fedora > repository. You'd only apply this module to solr docs that have a > field of :fedora_pid, for example. > > The current SolrDocument marc stuff could be applied using a module > as well. Because the current implementation is an OpenStruct though, > we loose the original ability to access fields using a strings or > symbols. For example, the standard solr doc coming from RSolr can be > used like doc['title'] or doc[:title]. If we were to use extend > instead, we'd still have that functionality. It could be done like > this: > > module SolrMARCDocument > > def marc > @marc ||= ( > reader = > MARC::XMLReader.new(StringIO.new(self.marc_display)).to_a > reader[0] > ) > end > > def marc_xml > self.marc.to_xml.to_s > > > end > > def to_xml > self.marc_xml > end > > end > Then: > > solr_doc.extend SolrMARCDocument > solr_doc.marc_xml > > So there wouldn't really be a concrete SolrDocument class. There > would be solr documents (hashes) with mixins applied to create > richer objects. The code that handles the "extend" could be in > SolrHelper. One method, that you could overridein your own app. > > ------------------ > > I guess what I'm saying is that we (could): > > - keep mapping/behaviors in the models (solr documents). We can do > this in a super elegant way by using standard ruby techniques > (extend, include). > > - keep the built-in BL behaviors as generic/simple as possible > > - have a single point in BL that has the opportunity to modify each > solr doc that Blacklight is going to use (SolrHelper) > > > Matt > > > 2009/3/31 Jamie Orchard-Hays > I see value in this, but I'm wondering if it's getting too clever. > Isn't the idea of having Blacklight as a plugin so that end users > can change the default functionality/behavior by overriding the > files in the RAILS_HOME/apps dir? Is this over-engineering the plugin? > > I'm not asking this rhetorically. I'm wondering. > > For example, if someone doesn't want our to_xml in SolrDocument, > can't they create or new SolrDocument in RAILS_HOME/apps/models/ and > have at it? (I'm unclear if the Rails Engine plugin allows changing > models this way.) > > Jamie > > > > > On Mar 31, 2009, at 10:07 PM, Matt Mitchell wrote: > >> I second the "good work"! >> >> For the config stuff, I've got some stuff I'm going to try and get >> in sometime this week. Using a standard hash, and some basic >> switching based on the RAILS_ENV value. >> >> Erik, I was thinking the same thing about the SolrDocument. What >> about solr documents that don't use any marc? There may be lots of >> different data types coming down the pipeline in the future. This >> is where inheritance will be a problem. >> >> The thing we did in the UVa implementation was to attach behaviors >> to solr documents, instead of wrapping them in another class. To do >> this, you can use "extend" on an object. Sorry for the code >> examples, but it's easier for me to explain... >> >> module MARCCommon >> # some common marc stuff here? >> end >> >> module MARC21Doc >> include MARCCommon >> def to_marc >> >> end >> end >> >> module MARCXmlDoc >> include MARCCommon >> def to_xml >> >> end >> end >> >> module DigitalLibraryDoc >> def images >> >> end >> end >> >> # in SolrHelper/controller, or some other custom "finder": >> docs = solr.select(:q=>'*:*') >> docs.each do |doc| >> doc.extend MARCXmlDoc if doc[:marc_xml] >> doc.extend MARC21Doc if doc[:marc_21] >> doc.extend DigitalLibraryDoc if doc[:digital_lib] >> end >> >> Later on in your view code (for example) >> >> @docs.each do |doc| >> if doc.kind_of?(DigitalLibraryDoc) >> puts 'I'm a digilib doc!' >> end >> end >> >> That way, each object can have it's own behavior, depending on its >> own characteristics. >> >> Matt >> >> On Tue, Mar 31, 2009 at 8:34 PM, Jessie Keck >> wrote: >> Hi all, >> >> Great job on getting release 2.0.0 out. Good work. We'll be >> upgrading in the next few days, I hope. >> >> So Naomi and I were looking at the SolrDocument class, and we >> realized a couple of things: >> >> 1. it seems like the id field name and the marc field name are >> hardcoded in there. >> 2. it seems like the code is set up only for marcxml. >> >> Naomi and I kicked around a few ideas. What do you think of this: >> >> a. adding storage_type as a property to solr.yml for "raw metadata >> type". Currently might only be marc21 and marcxml, but Vernon is >> interested in DC, and we'll be interested in MODS eventually ... >> >> b. have the SolrDocument initialize method check that property and >> initialize the class as appropriate. For example, marcxml uses the >> marc xmlreader, but marc21 uses the marc reader. DC might use >> something else, etc. >> >> >> I am currently working on this in line with my MLA/APA citation >> work. What I have done is added a new field to the solr.yml file >> and the DisplayFields module of storage_type (next to the >> marc_storage_field). Hopefully in the future the storage_type will >> be a solr field in the solr doc so that each record can be handled >> according to its data type. >> I am then doing a case statement on storage_type to instantiate the >> MARC::Record object from either MARC21 or MARC-XML. >> >> Then in the SolrDocument class's initialize method I am setting >> self.marc to that MARC::Record object so that the xml to marc_xml >> methods don't need any changes. >> >> Also, I plan on extending the SolrDocument class out a little more >> for my citations. I will have to_mla and to_apa methods of the >> SolrDocument object so that the citations can be easily called. >> >> I have working code written to do this. I will be extending the >> current SolrDocument spec to test both flavors before I move on >> (and I may need to ask for help with the spec). >> >> Thanks, >> -Jessie >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndushay at stanford.edu Wed Apr 1 13:36:13 2009 From: ndushay at stanford.edu (Naomi Dushay) Date: Wed, 1 Apr 2009 10:36:13 -0700 Subject: [Blacklight-development] HELP: solr_helper is ... where? Message-ID: <32E45A5C-10D7-4AAA-96A5-2AB591B7FAAA@stanford.edu> I can't find the solr_helper.rb class in trunk or in the 2.0.0 release. Where is it? I was looking in: rails/vendor/plugins/blacklight/app/helpers - Naomi From rochkind at jhu.edu Wed Apr 1 14:21:59 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Wed, 01 Apr 2009 14:21:59 -0400 Subject: [Blacklight-development] on the SolrDocument In-Reply-To: References: <49D2B69A.2010303@stanford.edu> Message-ID: <49D3B0C7.7040401@jhu.edu> "I'm on board with keeping the plugin generic ... but if we do that, we want to have exemplars." I may be a minority of one, but I still think the plugin should work out of the box for a newbie. You install the plugin, you install the MARC indexer, you point the marc indexer at a bunch of MARC files, and everything should Just Work with default assumptions. For everyone that wants to change/override the defaults, it should indeed be very easy to do that. But isn't sensible defaults that Just Work in the general case the ruby/rails way we like, as opposed to requiring certain configuration that a user may not understand to even get the thing working? I have that experience too often with efforts from the Java community, all this configuration I need to fill out before I can get the app to work at all, when I don't understand what any of the configuration even means because I'm a newcomer to the application. And again, the advantage of me in having these sensible defaults be in the plugin instead of 'generated', is that when a new version of the plug-in comes out, if the defaults are changed, or new neccesary defaults are added to support new features, and a local installation is using the defaults, then they get the new defaults _automatically_ just by upgrading the plugin. If these sensible defaults were instead in 'generated' code, then they've got to re-generate and merge in their customizations. Jonathan Naomi Dushay wrote: > This is a great discussion. It would be even better if we could all be together in a room with a whiteboard, but oh well. Great job on the emails with enough specifics to make a reasonable substitute. > > I'm on board with keeping the plugin generic ... but if we do that, we want to have exemplars. It would be great if Jessie and I were the only RoR newbs in potential blacklight adopters ... but I suspect we're not. And maybe I'm being Chicken Little here, but the more straightforward the code is, the better, for where I am currently. That would favor Matt's SolrDoc approach vs. ostruct. Maybe I'll retract that statement in the near future, but let's adhere to KISS whenever we can. (Of course, the crux of this is Jamie's interpretation of "simple" vs. "mine" :-) ) > > These exemplars demonstrating how to work with Marc21, Marcxml, DC, etc. might be part of the demo app, or there might be a couple of exemplar apps. We know we've got traction with marc exemplar. From what Jessie was saying, I think we only need "marc"; the distinction between marc21 and marcxml is just Marc::XMLReader vs. MARC::Reader in the initializer, AFAIK. But this is a quibble. Once Vernon codes for DC, hopefully we can grab his stuff as an exemplar as well. > > I see the to-be-entered JIRA tasks for refactoring as: > 1. how we initialize the code > 2. replacing solr.yml with overridable code > we also have > 3. cleanly splitting behaviors between plugin and exemplars. > > Thoughts? > > - Naomi > > On Apr 1, 2009, at 9:03 AM, Matt Mitchell wrote: > > We are talking about a few different things here I think. General app configuration, field mapping and domain issues. My thought for app configuration is that it could be a global, RAILS_ENV sensitive hash; Blacklight.config. The solr url and other configuration settings could be handled here using ruby. > > For display, we could chunk out as much as possible into partials. I think this is a good idea in general. Overriding files provides a high level approach to customizing and it's simple. Helper methods are another way, but can provide even more flexibility. Those are two things that Rails does out of the box and we should use them as much as possible without getting too complicated. So instead of: > > <%= DisplayFields.index_view[:show_link] %> > > Why not just: > > <%= show_link %> > > For the page title example that Naomi brought up, this could be a helper too: > > <%= doc_title %> > > The built-in, BL doc_title helper would have a little bit of logic if needed, and your app could override it too. And if done correctly, it could even override and call the built-in/super doc_title. > > Jamie, your question about just overriding a file in app/models... If you wanted to do it that way, you would create a new file at app/models/solr_document.rb. You would then have to call "require_dependency" to load the SolrDocument from the BL plugin. This would get you the original SolrDocument class definition. You'd then open up the class, and "include" another module that has the custom behavior. You can then call super on methods your module overrides. > > However, there seems to be a need for field mapping for the solr documents, stuff that's more in the model domain. I think putting this into the document model is the way to go for this, and not an external configuration mapping. My previous post might have sounded like I was saying BL should provide this functionality out-of-the-box; some kind of sub-framework for customizing solr-docs with "extend". That's not what I was suggesting though, that would definitely be over-engineering the plugin. > > I think that BL should limit the domain specific behavior if provides out of the box. And BL should provide an easy access point to when and where solr documents are created. Then specific institutions/implementations of BL could leverage this point to customize their models, using "extend" etc.. SolrHelper is the perfect spot for this. If we always use SolrHelper to create solr docs, then we can easily inject behavior into each document, and the rest of the app (partials, helpers) don't have to worry about it. > > Examples... > > There could be a BlacklightDisplayFields module that would be responsible for mapping generic BL fields like "title", "author". This module would intercept calls to those fields and map them to the correct solr field: > > module BlacklightDisplayFields > def title > self[:title_t] || 'Untitled' > end > def author > self[:author_t] > end > end > > solr_doc.extend BlacklightDisplayFields > > <%= solr_doc.title %> > > Another module would be responsible for querying an availablilty service. So, you'd only inject this module into solr docs that need this functionality, like library catalog items. > > One module could be responsible for providing links to a Fedora repository. You'd only apply this module to solr docs that have a field of :fedora_pid, for example. > > The current SolrDocument marc stuff could be applied using a module as well. Because the current implementation is an OpenStruct though, we loose the original ability to access fields using a strings or symbols. For example, the standard solr doc coming from RSolr can be used like doc['title'] or doc[:title]. If we were to use extend instead, we'd still have that functionality. It could be done like this: > > > module SolrMARCDocument > > def marc > @marc ||= ( > reader = MARC::XMLReader.new(StringIO.new(self.marc_display)).to_a > reader[0] > ) > end > > def marc_xml > self.marc.to_xml.to_s > > > end > > def to_xml > self.marc_xml > end > > end > > Then: > > solr_doc.extend SolrMARCDocument > solr_doc.marc_xml > > So there wouldn't really be a concrete SolrDocument class. There would be solr documents (hashes) with mixins applied to create richer objects. The code that handles the "extend" could be in SolrHelper. One method, that you could overridein your own app. > > ------------------ > > I guess what I'm saying is that we (could): > > - keep mapping/behaviors in the models (solr documents). We can do this in a super elegant way by using standard ruby techniques (extend, include). > > - keep the built-in BL behaviors as generic/simple as possible > > - have a single point in BL that has the opportunity to modify each solr doc that Blacklight is going to use (SolrHelper) > > > Matt > > > 2009/3/31 Jamie Orchard-Hays > > I see value in this, but I'm wondering if it's getting too clever. Isn't the idea of having Blacklight as a plugin so that end users can change the default functionality/behavior by overriding the files in the RAILS_HOME/apps dir? Is this over-engineering the plugin? > > I'm not asking this rhetorically. I'm wondering. > > For example, if someone doesn't want our to_xml in SolrDocument, can't they create or new SolrDocument in RAILS_HOME/apps/models/ and have at it? (I'm unclear if the Rails Engine plugin allows changing models this way.) > > Jamie > > > > > On Mar 31, 2009, at 10:07 PM, Matt Mitchell wrote: > > I second the "good work"! > > For the config stuff, I've got some stuff I'm going to try and get in sometime this week. Using a standard hash, and some basic switching based on the RAILS_ENV value. > > Erik, I was thinking the same thing about the SolrDocument. What about solr documents that don't use any marc? There may be lots of different data types coming down the pipeline in the future. This is where inheritance will be a problem. > > The thing we did in the UVa implementation was to attach behaviors to solr documents, instead of wrapping them in another class. To do this, you can use "extend" on an object. Sorry for the code examples, but it's easier for me to explain... > > module MARCCommon > # some common marc stuff here? > end > > module MARC21Doc > include MARCCommon > def to_marc > > end > end > > module MARCXmlDoc > include MARCCommon > def to_xml > > end > end > > module DigitalLibraryDoc > def images > > end > end > > # in SolrHelper/controller, or some other custom "finder": > docs = solr.select(:q=>'*:*') > docs.each do |doc| > doc.extend MARCXmlDoc if doc[:marc_xml] > doc.extend MARC21Doc if doc[:marc_21] > doc.extend DigitalLibraryDoc if doc[:digital_lib] > end > > Later on in your view code (for example) > > @docs.each do |doc| > if doc.kind_of?(DigitalLibraryDoc) > puts 'I'm a digilib doc!' > end > end > > That way, each object can have it's own behavior, depending on its own characteristics. > > Matt > > On Tue, Mar 31, 2009 at 8:34 PM, Jessie Keck > wrote: > Hi all, > > Great job on getting release 2.0.0 out. Good work. We'll be upgrading in the next few days, I hope. > > So Naomi and I were looking at the SolrDocument class, and we realized a couple of things: > > 1. it seems like the id field name and the marc field name are hardcoded in there. > 2. it seems like the code is set up only for marcxml. > > Naomi and I kicked around a few ideas. What do you think of this: > > a. adding storage_type as a property to solr.yml for "raw metadata type". Currently might only be marc21 and marcxml, but Vernon is interested in DC, and we'll be interested in MODS eventually ... > > b. have the SolrDocument initialize method check that property and initialize the class as appropriate. For example, marcxml uses the marc xmlreader, but marc21 uses the marc reader. DC might use something else, etc. > > > I am currently working on this in line with my MLA/APA citation work. What I have done is added a new field to the solr.yml file and the DisplayFields module of storage_type (next to the marc_storage_field). Hopefully in the future the storage_type will be a solr field in the solr doc so that each record can be handled according to its data type. > I am then doing a case statement on storage_type to instantiate the MARC::Record object from either MARC21 or MARC-XML. > > Then in the SolrDocument class's initialize method I am setting self.marc to that MARC::Record object so that the xml to marc_xml methods don't need any changes. > > Also, I plan on extending the SolrDocument class out a little more for my citations. I will have to_mla and to_apa methods of the SolrDocument object so that the citations can be easily called. > > I have working code written to do this. I will be extending the current SolrDocument spec to test both flavors before I move on (and I may need to ask for help with the spec). > > Thanks, > -Jessie > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > From ndushay at stanford.edu Wed Apr 1 14:50:54 2009 From: ndushay at stanford.edu (Naomi Dushay) Date: Wed, 1 Apr 2009 11:50:54 -0700 Subject: [Blacklight-development] on the SolrDocument In-Reply-To: <49D3B0C7.7040401@jhu.edu> References: <49D2B69A.2010303@stanford.edu> <49D3B0C7.7040401@jhu.edu> Message-ID: Think exemplar. We need a marc-solrmarc-to-populate exemplar, as that will be a significant community. The exemplar should be easy to install and get running. There will be documentation to indicate how you can modify it. On Apr 1, 2009, at 11:21 AM, Jonathan Rochkind wrote: > "I'm on board with keeping the plugin generic ... but if we do that, > we want to have exemplars." > > I may be a minority of one, but I still think the plugin should work > out of the box for a newbie. You install the plugin, you install > the MARC indexer, you point the marc indexer at a bunch of MARC > files, and everything should Just Work with default assumptions. > > For everyone that wants to change/override the defaults, it should > indeed be very easy to do that. > > But isn't sensible defaults that Just Work in the general case the > ruby/rails way we like, as opposed to requiring certain > configuration that a user may not understand to even get the thing > working? I have that experience too often with efforts from the > Java community, all this configuration I need to fill out before I > can get the app to work at all, when I don't understand what any of > the configuration even means because I'm a newcomer to the > application. > > And again, the advantage of me in having these sensible defaults be > in the plugin instead of 'generated', is that when a new version of > the plug-in comes out, if the defaults are changed, or new neccesary > defaults are added to support new features, and a local installation > is using the defaults, then they get the new defaults > _automatically_ just by upgrading the plugin. If these sensible > defaults were instead in 'generated' code, then they've got to re- > generate and merge in their customizations. > > Jonathan > > Naomi Dushay wrote: >> This is a great discussion. It would be even better if we could >> all be together in a room with a whiteboard, but oh well. Great >> job on the emails with enough specifics to make a reasonable >> substitute. >> >> I'm on board with keeping the plugin generic ... but if we do that, >> we want to have exemplars. It would be great if Jessie and I were >> the only RoR newbs in potential blacklight adopters ... but I >> suspect we're not. And maybe I'm being Chicken Little here, but >> the more straightforward the code is, the better, for where I am >> currently. That would favor Matt's SolrDoc approach vs. ostruct. >> Maybe I'll retract that statement in the near future, but let's >> adhere to KISS whenever we can. (Of course, the crux of this is >> Jamie's interpretation of "simple" vs. "mine" :-) ) >> >> These exemplars demonstrating how to work with Marc21, Marcxml, DC, >> etc. might be part of the demo app, or there might be a couple of >> exemplar apps. We know we've got traction with marc exemplar. >> From what Jessie was saying, I think we only need "marc"; the >> distinction between marc21 and marcxml is just Marc::XMLReader >> vs. MARC::Reader in the initializer, AFAIK. But this is a >> quibble. Once Vernon codes for DC, hopefully we can grab his stuff >> as an exemplar as well. >> >> I see the to-be-entered JIRA tasks for refactoring as: >> 1. how we initialize the code >> 2. replacing solr.yml with overridable code >> we also have >> 3. cleanly splitting behaviors between plugin and exemplars. >> >> Thoughts? >> >> - Naomi >> >> On Apr 1, 2009, at 9:03 AM, Matt Mitchell wrote: >> >> We are talking about a few different things here I think. General >> app configuration, field mapping and domain issues. My thought for >> app configuration is that it could be a global, RAILS_ENV sensitive >> hash; Blacklight.config. The solr url and other configuration >> settings could be handled here using ruby. >> >> For display, we could chunk out as much as possible into partials. >> I think this is a good idea in general. Overriding files provides a >> high level approach to customizing and it's simple. Helper methods >> are another way, but can provide even more flexibility. Those are >> two things that Rails does out of the box and we should use them as >> much as possible without getting too complicated. So instead of: >> >> <%= DisplayFields.index_view[:show_link] %> >> >> Why not just: >> >> <%= show_link %> >> >> For the page title example that Naomi brought up, this could be a >> helper too: >> >> <%= doc_title %> >> >> The built-in, BL doc_title helper would have a little bit of logic >> if needed, and your app could override it too. And if done >> correctly, it could even override and call the built-in/super >> doc_title. >> >> Jamie, your question about just overriding a file in app/models... >> If you wanted to do it that way, you would create a new file at app/ >> models/solr_document.rb. You would then have to call >> "require_dependency" to load the SolrDocument from the BL plugin. >> This would get you the original SolrDocument class definition. >> You'd then open up the class, and "include" another module that has >> the custom behavior. You can then call super on methods your module >> overrides. >> >> However, there seems to be a need for field mapping for the solr >> documents, stuff that's more in the model domain. I think putting >> this into the document model is the way to go for this, and not an >> external configuration mapping. My previous post might have sounded >> like I was saying BL should provide this functionality out-of-the- >> box; some kind of sub-framework for customizing solr-docs with >> "extend". That's not what I was suggesting though, that would >> definitely be over-engineering the plugin. >> >> I think that BL should limit the domain specific behavior if >> provides out of the box. And BL should provide an easy access point >> to when and where solr documents are created. Then specific >> institutions/implementations of BL could leverage this point to >> customize their models, using "extend" etc.. SolrHelper is the >> perfect spot for this. If we always use SolrHelper to create solr >> docs, then we can easily inject behavior into each document, and >> the rest of the app (partials, helpers) don't have to worry about it. >> >> Examples... >> >> There could be a BlacklightDisplayFields module that would be >> responsible for mapping generic BL fields like "title", "author". >> This module would intercept calls to those fields and map them to >> the correct solr field: >> >> module BlacklightDisplayFields >> def title >> self[:title_t] || 'Untitled' >> end >> def author >> self[:author_t] >> end >> end >> >> solr_doc.extend BlacklightDisplayFields >> >> <%= solr_doc.title %> >> >> Another module would be responsible for querying an availablilty >> service. So, you'd only inject this module into solr docs that need >> this functionality, like library catalog items. >> >> One module could be responsible for providing links to a Fedora >> repository. You'd only apply this module to solr docs that have a >> field of :fedora_pid, for example. >> >> The current SolrDocument marc stuff could be applied using a module >> as well. Because the current implementation is an OpenStruct >> though, we loose the original ability to access fields using a >> strings or symbols. For example, the standard solr doc coming from >> RSolr can be used like doc['title'] or doc[:title]. If we were to >> use extend instead, we'd still have that functionality. It could be >> done like this: >> >> >> module SolrMARCDocument >> >> def marc >> @marc ||= ( >> reader = >> MARC::XMLReader.new(StringIO.new(self.marc_display)).to_a >> reader[0] >> ) >> end >> >> def marc_xml >> self.marc.to_xml.to_s >> >> >> end >> >> def to_xml >> self.marc_xml >> end >> >> end >> >> Then: >> >> solr_doc.extend SolrMARCDocument >> solr_doc.marc_xml >> >> So there wouldn't really be a concrete SolrDocument class. There >> would be solr documents (hashes) with mixins applied to create >> richer objects. The code that handles the "extend" could be in >> SolrHelper. One method, that you could overridein your own app. >> >> ------------------ >> >> I guess what I'm saying is that we (could): >> >> - keep mapping/behaviors in the models (solr documents). We can do >> this in a super elegant way by using standard ruby techniques >> (extend, include). >> >> - keep the built-in BL behaviors as generic/simple as possible >> >> - have a single point in BL that has the opportunity to modify each >> solr doc that Blacklight is going to use (SolrHelper) >> >> >> Matt >> >> >> 2009/3/31 Jamie Orchard-Hays > >> I see value in this, but I'm wondering if it's getting too clever. >> Isn't the idea of having Blacklight as a plugin so that end users >> can change the default functionality/behavior by overriding the >> files in the RAILS_HOME/apps dir? Is this over-engineering the >> plugin? >> >> I'm not asking this rhetorically. I'm wondering. >> >> For example, if someone doesn't want our to_xml in SolrDocument, >> can't they create or new SolrDocument in RAILS_HOME/apps/models/ >> and have at it? (I'm unclear if the Rails Engine plugin allows >> changing models this way.) >> >> Jamie >> >> >> >> >> On Mar 31, 2009, at 10:07 PM, Matt Mitchell wrote: >> >> I second the "good work"! >> >> For the config stuff, I've got some stuff I'm going to try and get >> in sometime this week. Using a standard hash, and some basic >> switching based on the RAILS_ENV value. >> >> Erik, I was thinking the same thing about the SolrDocument. What >> about solr documents that don't use any marc? There may be lots of >> different data types coming down the pipeline in the future. This >> is where inheritance will be a problem. >> >> The thing we did in the UVa implementation was to attach behaviors >> to solr documents, instead of wrapping them in another class. To do >> this, you can use "extend" on an object. Sorry for the code >> examples, but it's easier for me to explain... >> >> module MARCCommon >> # some common marc stuff here? >> end >> >> module MARC21Doc >> include MARCCommon >> def to_marc >> >> end >> end >> >> module MARCXmlDoc >> include MARCCommon >> def to_xml >> >> end >> end >> >> module DigitalLibraryDoc >> def images >> >> end >> end >> >> # in SolrHelper/controller, or some other custom "finder": >> docs = solr.select(:q=>'*:*') >> docs.each do |doc| >> doc.extend MARCXmlDoc if doc[:marc_xml] >> doc.extend MARC21Doc if doc[:marc_21] >> doc.extend DigitalLibraryDoc if doc[:digital_lib] >> end >> >> Later on in your view code (for example) >> >> @docs.each do |doc| >> if doc.kind_of?(DigitalLibraryDoc) >> puts 'I'm a digilib doc!' >> end >> end >> >> That way, each object can have it's own behavior, depending on its >> own characteristics. >> >> Matt >> >> On Tue, Mar 31, 2009 at 8:34 PM, Jessie Keck > >> wrote: >> Hi all, >> >> Great job on getting release 2.0.0 out. Good work. We'll be >> upgrading in the next few days, I hope. >> >> So Naomi and I were looking at the SolrDocument class, and we >> realized a couple of things: >> >> 1. it seems like the id field name and the marc field name are >> hardcoded in there. >> 2. it seems like the code is set up only for marcxml. >> >> Naomi and I kicked around a few ideas. What do you think of this: >> >> a. adding storage_type as a property to solr.yml for "raw metadata >> type". Currently might only be marc21 and marcxml, but Vernon is >> interested in DC, and we'll be interested in MODS eventually ... >> >> b. have the SolrDocument initialize method check that property and >> initialize the class as appropriate. For example, marcxml uses the >> marc xmlreader, but marc21 uses the marc reader. DC might use >> something else, etc. >> >> >> I am currently working on this in line with my MLA/APA citation >> work. What I have done is added a new field to the solr.yml file >> and the DisplayFields module of storage_type (next to the >> marc_storage_field). Hopefully in the future the storage_type will >> be a solr field in the solr doc so that each record can be handled >> according to its data type. >> I am then doing a case statement on storage_type to instantiate the >> MARC::Record object from either MARC21 or MARC-XML. >> >> Then in the SolrDocument class's initialize method I am setting >> self.marc to that MARC::Record object so that the xml to marc_xml >> methods don't need any changes. >> >> Also, I plan on extending the SolrDocument class out a little more >> for my citations. I will have to_mla and to_apa methods of the >> SolrDocument object so that the citations can be easily called. >> >> I have working code written to do this. I will be extending the >> current SolrDocument spec to test both flavors before I move on >> (and I may need to ask for help with the spec). >> >> Thanks, >> -Jessie >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org> > >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org> > >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org> > >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org> > >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ >> >> >> > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From jamie at dang.com Wed Apr 1 15:51:33 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Wed, 1 Apr 2009 15:51:33 -0400 Subject: [Blacklight-development] on the SolrDocument In-Reply-To: References: <49D2B69A.2010303@stanford.edu> Message-ID: <41FE2A64-E137-4BE3-909C-0686F40FB40B@dang.com> This is not true. When Bess and I introduced the SolrDocument we gave it a [] method that will take the String or Symbol equivalent of the attribute. As a matter of fact, the way it's implemented, you can access the attribute with either a String such as "foo" or a Symbol such as :foo. On Apr 1, 2009, at 12:03 PM, Matt Mitchell wrote: > Because the current implementation is an OpenStruct though, we loose > the original ability to access fields using a strings or symbols. From jamie at dang.com Wed Apr 1 16:05:50 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Wed, 1 Apr 2009 16:05:50 -0400 Subject: [Blacklight-development] Blacklight Mailing list Message-ID: <0CD8183E-5886-4B98-ACFB-314E06B9D3D7@dang.com> Is anyone else feeling the pain of the slowness of mail delivery on this list? It often takes nearly an hour for posts to get mailed (they show up in the archive immediately). Sometimes newer posts arrive before older posts. I've written Tom (one of the admins of Rubyforge) but haven't heard back yet. Jamie From jamie at dang.com Wed Apr 1 15:47:03 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Wed, 1 Apr 2009 15:47:03 -0400 Subject: [Blacklight-development] Overriding SolrDocument In-Reply-To: References: <49D2B69A.2010303@stanford.edu> Message-ID: Ew. I don't think that passes the KISS test. On Apr 1, 2009, at 12:03 PM, Matt Mitchell wrote: > Jamie, your question about just overriding a file in app/models... > If you wanted to do it that way, you would create a new file at app/ > models/solr_document.rb. You would then have to call > "require_dependency" to load the SolrDocument from the BL plugin. > This would get you the original SolrDocument class definition. You'd > then open up the class, and "include" another module that has the > custom behavior. You can then call super on methods your module > overrides. From rochkind at jhu.edu Wed Apr 1 16:24:31 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Wed, 01 Apr 2009 16:24:31 -0400 Subject: [Blacklight-development] Blacklight Mailing list In-Reply-To: <0CD8183E-5886-4B98-ACFB-314E06B9D3D7@dang.com> References: <0CD8183E-5886-4B98-ACFB-314E06B9D3D7@dang.com> Message-ID: <49D3CD7F.5000803@jhu.edu> Yes. It's not just slow, but unpredictable. Sometimes I send two posts, and my second arrives after my first. VERY odd. Jamie Orchard-Hays wrote: > Is anyone else feeling the pain of the slowness of mail delivery on > this list? It often takes nearly an hour for posts to get mailed (they > show up in the archive immediately). Sometimes newer posts arrive > before older posts. > > I've written Tom (one of the admins of Rubyforge) but haven't heard > back yet. > > Jamie > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From chapman.lists at gmail.com Wed Apr 1 16:23:09 2009 From: chapman.lists at gmail.com (Vernon Chapman) Date: Wed, 01 Apr 2009 16:23:09 -0400 Subject: [Blacklight-development] HELP: solr_helper is ... where? In-Reply-To: <32E45A5C-10D7-4AAA-96A5-2AB591B7FAAA@stanford.edu> Message-ID: Naomi, I'm not sure if anyone answered you question but the solr_helper.rb is now in vendor/plugins/blacklight/lib/blacklight Vern On 4/1/09 1:36 PM, "Naomi Dushay" wrote: > I can't find the solr_helper.rb class in trunk or in the 2.0.0 > release. Where is it? I was looking in: > > rails/vendor/plugins/blacklight/app/helpers > > - Naomi > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From goodieboy at gmail.com Wed Apr 1 16:33:40 2009 From: goodieboy at gmail.com (Matt Mitchell) Date: Wed, 1 Apr 2009 16:33:40 -0400 Subject: [Blacklight-development] Overriding SolrDocument In-Reply-To: References: <49D2B69A.2010303@stanford.edu> Message-ID: Yeah, I'm with you. I don't like it either. Just stating what you *could* do. On Wed, Apr 1, 2009 at 3:47 PM, Jamie Orchard-Hays wrote: > Ew. I don't think that passes the KISS test. > > > On Apr 1, 2009, at 12:03 PM, Matt Mitchell wrote: > > Jamie, your question about just overriding a file in app/models... If you >> wanted to do it that way, you would create a new file at >> app/models/solr_document.rb. You would then have to call >> "require_dependency" to load the SolrDocument from the BL plugin. This would >> get you the original SolrDocument class definition. You'd then open up the >> class, and "include" another module that has the custom behavior. You can >> then call super on methods your module overrides. >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From goodieboy at gmail.com Wed Apr 1 16:29:48 2009 From: goodieboy at gmail.com (Matt Mitchell) Date: Wed, 1 Apr 2009 16:29:48 -0400 Subject: [Blacklight-development] on the SolrDocument In-Reply-To: <41FE2A64-E137-4BE3-909C-0686F40FB40B@dang.com> References: <49D2B69A.2010303@stanford.edu> <41FE2A64-E137-4BE3-909C-0686F40FB40B@dang.com> Message-ID: Sorry Jamie, didn't see that :) On Wed, Apr 1, 2009 at 3:51 PM, Jamie Orchard-Hays wrote: > This is not true. When Bess and I introduced the SolrDocument we gave it a > [] method that will take the String or Symbol equivalent of the attribute. > As a matter of fact, the way it's implemented, you can access the attribute > with either a String such as "foo" or a Symbol such as :foo. > > > On Apr 1, 2009, at 12:03 PM, Matt Mitchell wrote: > > Because the current implementation is an OpenStruct though, we loose the >> original ability to access fields using a strings or symbols. >> > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamie at dang.com Wed Apr 1 19:23:20 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Wed, 1 Apr 2009 19:23:20 -0400 Subject: [Blacklight-development] on the SolrDocument In-Reply-To: References: <49D2B69A.2010303@stanford.edu> <41FE2A64-E137-4BE3-909C-0686F40FB40B@dang.com> Message-ID: <8726E283-83D7-4528-98C8-8808DAA7E47E@dang.com> Wow, this message took 3 hours to be delivered... Google groups anyone? :-D Jamie On Apr 1, 2009, at 4:29 PM, Matt Mitchell wrote: > Sorry Jamie, didn't see that :) > > On Wed, Apr 1, 2009 at 3:51 PM, Jamie Orchard-Hays > wrote: > This is not true. When Bess and I introduced the SolrDocument we > gave it a [] method that will take the String or Symbol equivalent > of the attribute. As a matter of fact, the way it's implemented, you > can access the attribute with either a String such as "foo" or a > Symbol such as :foo. > > > > On Apr 1, 2009, at 12:03 PM, Matt Mitchell wrote: > > Because the current implementation is an OpenStruct though, we loose > the original ability to access fields using a strings or symbols. > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eos8d at virginia.edu Thu Apr 2 10:31:55 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Thu, 2 Apr 2009 10:31:55 -0400 Subject: [Blacklight-development] packaging solrmarc with blacklight-2.1 Message-ID: Hey, folks. It seems to me that one of our next steps needs to be packaging solrmarc with our blacklight distro. Stanford and UVA are both using solrmarc to do their indexing, and the ruby indexer that currently comes with the plugin was only ever meant to be a placeholder. I'm volunteering to do this work, especially since I have Bob Haschart right here in the building with me to help if I get stuck. It seems to me we should distribute a copy of solrmarc with our blacklight distro, in keeping with our "it should work out of the box" goals. Then we can have documentation about how to compile with the default settings, how to make changes to the config and re-compile, and how to set it loose on your own data. Sound good? Bess From rochkind at jhu.edu Thu Apr 2 11:34:53 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 02 Apr 2009 11:34:53 -0400 Subject: [Blacklight-development] packaging solrmarc with blacklight-2.1 In-Reply-To: References: Message-ID: <49D4DB1D.4010804@jhu.edu> Awesome! Thanks Bess, I agree entirely. Bess Sadler wrote: > Hey, folks. > > It seems to me that one of our next steps needs to be packaging > solrmarc with our blacklight distro. Stanford and UVA are both using > solrmarc to do their indexing, and the ruby indexer that currently > comes with the plugin was only ever meant to be a placeholder. I'm > volunteering to do this work, especially since I have Bob Haschart > right here in the building with me to help if I get stuck. > > It seems to me we should distribute a copy of solrmarc with our > blacklight distro, in keeping with our "it should work out of the box" > goals. Then we can have documentation about how to compile with the > default settings, how to make changes to the config and re-compile, > and how to set it loose on your own data. > > Sound good? > > Bess > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From ndushay at stanford.edu Thu Apr 2 12:52:25 2009 From: ndushay at stanford.edu (Naomi Dushay) Date: Thu, 2 Apr 2009 09:52:25 -0700 Subject: [Blacklight-development] packaging solrmarc with blacklight-2.1 In-Reply-To: References: Message-ID: <41400396-2341-4643-B892-67D13A7FCCC9@stanford.edu> Bess, this sounds great. We also need to set up the solr_helper tests so that, from within solr_helper_spec, we are calling out to solrmarc to index known data with a known schema and solrconfig. - Naomi On Apr 2, 2009, at 7:31 AM, Bess Sadler wrote: > Hey, folks. > > It seems to me that one of our next steps needs to be packaging > solrmarc with our blacklight distro. Stanford and UVA are both using > solrmarc to do their indexing, and the ruby indexer that currently > comes with the plugin was only ever meant to be a placeholder. I'm > volunteering to do this work, especially since I have Bob Haschart > right here in the building with me to help if I get stuck. > > It seems to me we should distribute a copy of solrmarc with our > blacklight distro, in keeping with our "it should work out of the > box" goals. Then we can have documentation about how to compile with > the default settings, how to make changes to the config and re- > compile, and how to set it loose on your own data. > > Sound good? > > Bess > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From eos8d at virginia.edu Thu Apr 2 13:03:48 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Thu, 2 Apr 2009 13:03:48 -0400 Subject: [Blacklight-development] packaging solrmarc with blacklight-2.1 In-Reply-To: <41400396-2341-4643-B892-67D13A7FCCC9@stanford.edu> References: <41400396-2341-4643-B892-67D13A7FCCC9@stanford.edu> Message-ID: Yes, I totally agree. I'll work on that, too. Bess On 2-Apr-09, at 12:52 PM, Naomi Dushay wrote: > Bess, this sounds great. > > We also need to set up the solr_helper tests so that, from within > solr_helper_spec, we are calling out to solrmarc to index known data > with a known schema and solrconfig. > > - Naomi > > On Apr 2, 2009, at 7:31 AM, Bess Sadler wrote: > >> Hey, folks. >> >> It seems to me that one of our next steps needs to be packaging >> solrmarc with our blacklight distro. Stanford and UVA are both using >> solrmarc to do their indexing, and the ruby indexer that currently >> comes with the plugin was only ever meant to be a placeholder. I'm >> volunteering to do this work, especially since I have Bob Haschart >> right here in the building with me to help if I get stuck. >> >> It seems to me we should distribute a copy of solrmarc with our >> blacklight distro, in keeping with our "it should work out of the >> box" goals. Then we can have documentation about how to compile with >> the default settings, how to make changes to the config and re- >> compile, and how to set it loose on your own data. >> >> Sound good? >> >> Bess >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From jkeck at stanford.edu Thu Apr 2 19:26:24 2009 From: jkeck at stanford.edu (Jessie Keck) Date: Thu, 02 Apr 2009 16:26:24 -0700 Subject: [Blacklight-development] on the SolrDocument In-Reply-To: <97CFBC79-2E23-48D2-A3B7-CF3F7F671448@mac.com> References: <49D2B69A.2010303@stanford.edu> <5836B658-2589-4C50-8DE5-BB3DF2C1CA35@stanford.edu> <97CFBC79-2E23-48D2-A3B7-CF3F7F671448@mac.com> Message-ID: <49D549A0.4010302@stanford.edu> This may be coming completely out of left field, but when you said a true object model, it got me thinking a little bit. What if we had a database backed administration that would allow for the creation of Configuration objects with a format attribute? So you may have a configuration table that would look something like this: {:id =>2, :format => 'default', :index_view_fields => ['title_t', 'author_t', 'format_facet'], :show_view_fields => ['title_t', 'format_facet', etc....]} {:id =>2, :format => 'book', :index_view_fields => ['title_t', 'author_t', 'format_facet'], :show_view_fields => ['title_t', 'format_facet', etc....]} {:id =>2, :format => 'music', :index_view_fields => ['title_t', 'composer_t'], :show_view_fields => ['title_t', 'composer_t', etc.....]} We could then do something like: fields = Configuration.find_by_format(document.format) <% fields.show_view_fields.each do |field| %>
<%= field_label field %>
<%= document.get field %>
<% end %> This would allow for views configurations for several different formats. This would take away the need for _index and _show partials for the various content types. I'm not saying this is a better alternative, this is just a thought. We're already using configurations from YAML files to try and kludge an configuration object, why not do it the right way? (In the future of course). As I see it, down the road we are going to need to expand all of these type of configurations one way or the other. Obviously the easiest way from a development standpoint is to just say, "Override the template". But if we're able to minimize the number of templates that need to overridden just to get the app running and allow for easier configurations of display fields w/o the user having an intimate knowledge of Blacklight and/or Rails the better. I think that could make Blacklight a more attractive application. This is just a thought way out into the future and may be completely wrong, but I thought I would see if anybody has interesting thoughts on this as being a possible alternative to the configuration discussions we've been having. Thanks, -Jessie Keck Erik Hatcher wrote: > > On Mar 31, 2009, at 10:34 PM, Naomi Dushay wrote: >> I wish I understood this better. What you're saying about solr.yml >> makes some sense to me - it's becoming a real hodge-podge of >> information meant as a sort of abstraction of some basic concepts, >> and it keeps growing. > > Basically what I'm saying is to *model* this stuff as a true _object > model_, not config files. Let the customizations and such be tweaks > to a real object model, not hideous looking fat YAML files. > >> This all started because SOLR field names were hardcoded into a >> variety of places: views, helpers, blah blah. That seemed like a >> bad idea - silly to constrain blacklight users to have specific field >> names in their SOLR index to match specific blacklight functionality. > > Definitely an abstraction is needed, no question. I dig that. > >> Maybe I could understand your suggestion better if you could explain >> it a little more, in the context of one or more of our specific needs. > > Sure, just some thoughts thrown out here.... > >> >> >> Here is the type of information we're trying to make easily >> configurable: >> >> individual item view: >> html title > > HTML title sounds like an override of a template, or partial. But > could you elaborate on what difference in item title UVa wants and > Stanford wants? > >> heading (most prominent text pertinent to individual item on the page) > > Again, a template override. > >> what data to display > > Again, a template override. This likely means that the detail > template should be broken into several smaller aggregated partial > templates that can be overridden leaner and cleaner. > > Although this does perhaps lend itself to having the fields listed in > a config file. But there could be many different types of items that > need different configurations - could get messy quickly. Doing this > at the template level and working hard on having as much common as > possible seems the first steps though. > >> search results view: >> heading data for each item >> what data will be displayed > > Again, template overrides for these. > >> facets: >> how to label them >> how to order them > > Trickier business for these. But this could be done in solrconfig.xml > - labels come from &facet.field={!key=YourCustomLabel}subject_facet, > and of course ordering them can be specified there as well. > > There's other things to consider here... like is it better to have > different search views (like UVa's general catalog vs. music) defined > as separate request handlers in solrconfig.xml? Or should that > configuration be more dynamically magically defined in Ruby? Or > perhaps a little hybrid where there is an object model that defines > these things and then a solrconfig.xml is generated. > >> Currently, we seem to be comfortable with the expectation that >> blacklight will run on top of SOLR. > > Blacklight without Solr would be, well.... mighty dark :) > >> We know it won't always be marc. We know field names will vary. We >> know different institutions will be writing code at the app level to >> override lots of stuff: look & feel, various behaviors, whatever. > > Right on, and I totally dig that. I just don't think a .yml file is > how you want the bulk of these overrides/customizations to occur. > >> How can we make it easy for blacklight adopters to get up and >> running, yet maximize customizability with a minimum of burden (for >> those installations that are okay with a fairly vanilla approach)? >> We also want to make the plugin feature rich without shoving >> requirements down adopters throats. > > I think you're on the right track... but rethinking how this config > stuff works is the main point here. > > Erik > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From jamie at dang.com Fri Apr 3 01:22:14 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Fri, 3 Apr 2009 01:22:14 -0400 Subject: [Blacklight-development] interesting post on Rails configuration Message-ID: http://blog.stonean.com/2009/03/06/convention-of-configuration/ From rochkind at jhu.edu Fri Apr 3 10:25:20 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Fri, 03 Apr 2009 10:25:20 -0400 Subject: [Blacklight-development] interesting post on Rails configuration In-Reply-To: References: Message-ID: <49D61C50.2090305@jhu.edu> The Rails 2.x initializers are also a good place for configuration, and they are also actual ruby code. My impression was that in Rails 2.x you were encouraged to use initializers instead of environment.rb. But it ends up working pretty much the same either way, just a question of what makes it easier to keep your code organized. I like initializers. Jamie Orchard-Hays wrote: > http://blog.stonean.com/2009/03/06/convention-of-configuration/ > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From eos8d at virginia.edu Fri Apr 3 11:17:56 2009 From: eos8d at virginia.edu (Bess Sadler) Date: Fri, 3 Apr 2009 11:17:56 -0400 Subject: [Blacklight-development] cucumber is my brand new pair of rollerskates Message-ID: I somehow got the impression that Jamie wanted to hear our impressions of cucumber. I am really liking it so far. I went through a couple of the online tutorials and found myself wanting more documentation, so I went out and bought The Rspec Book PDF (http://www.pragprog.com/titles/achbd/the-rspec-book ) and it seems pretty great too. I like cucumber because I wanted some high-level acceptance tests for VirgoBeta. So, for example, we get bug reports like this: When I did a search in Mozilla Firefox for "united states in the air" the title comes up on the first page in VB, but toward bottom of page. The title, United States in the Air, is exact. Can relevancy be improved? It's been quite frustrating to fix some of these issues and then have them re-appear later because we changed something else that seemed unrelated. So now I have a cucumber feature like this: Feature: Find Relevant Items In order to find relevant items As a user I want to see my expected items on the first screen Scenario: united states in the air Given a blank search When I search for "united states in the air" Then I should see record "u4823821" first This makes a lot more sense to me than unit tests. I'm not saying unit tests aren't important, of course they are. But being able to test the application as a whole seems like something I've been missing, and I think it's going to help me respond to users and more easily keep our application behaving the way they expect. Bess From goodieboy at gmail.com Fri Apr 3 11:47:55 2009 From: goodieboy at gmail.com (Matt Mitchell) Date: Fri, 3 Apr 2009 11:47:55 -0400 Subject: [Blacklight-development] cucumber is my brand new pair of rollerskates In-Reply-To: References: Message-ID: That's pretty nifty actually. I'm in the process of thinking about how to re-write the RSolr tests. I want more coverage and have reading about Shoulda. I plan on checking out RSpec more too. But Shoulda seems like a nice simple way to do BDD and feels more familiar than RSpec. I wonder if cucumber is compatible with Shoulda? OK, on to watch that cucumber video! Matt On Fri, Apr 3, 2009 at 11:17 AM, Bess Sadler wrote: > I somehow got the impression that Jamie wanted to hear our impressions of > cucumber. I am really liking it so far. I went through a couple of the > online tutorials and found myself wanting more documentation, so I went out > and bought The Rspec Book PDF ( > http://www.pragprog.com/titles/achbd/the-rspec-book) and it seems pretty > great too. > > I like cucumber because I wanted some high-level acceptance tests for > VirgoBeta. So, for example, we get bug reports like this: > > When I did a search in Mozilla Firefox for "united states in the air" the > title comes up on the first page in VB, but toward bottom of page. The > title, United States in the Air, is exact. Can relevancy be improved? > > It's been quite frustrating to fix some of these issues and then have them > re-appear later because we changed something else that seemed unrelated. So > now I have a cucumber feature like this: > > Feature: Find Relevant Items > In order to find relevant items > As a user > I want to see my expected items on the first screen > > Scenario: united states in the air > Given a blank search > When I search for "united states in the air" > Then I should see record "u4823821" first > > This makes a lot more sense to me than unit tests. I'm not saying unit > tests aren't important, of course they are. But being able to test the > application as a whole seems like something I've been missing, and I think > it's going to help me respond to users and more easily keep our application > behaving the way they expect. > > Bess > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erikhatcher at mac.com Fri Apr 3 12:00:55 2009 From: erikhatcher at mac.com (Erik Hatcher) Date: Fri, 03 Apr 2009 12:00:55 -0400 Subject: [Blacklight-development] cucumber is my brand new pair of rollerskates In-Reply-To: References: Message-ID: <2C2440AD-755E-42FE-9174-11EAE8D31D41@mac.com> +1 The more testing the better, and everyone should feel empowered to test things in various ways. Units all the way up. Erik On Apr 3, 2009, at 11:17 AM, Bess Sadler wrote: > I somehow got the impression that Jamie wanted to hear our > impressions of cucumber. I am really liking it so far. I went > through a couple of the online tutorials and found myself wanting > more documentation, so I went out and bought The Rspec Book PDF (http://www.pragprog.com/titles/achbd/the-rspec-book > ) and it seems pretty great too. > > I like cucumber because I wanted some high-level acceptance tests > for VirgoBeta. So, for example, we get bug reports like this: > > When I did a search in Mozilla Firefox for "united states in the > air" the title comes up on the first page in VB, but toward bottom > of page. The title, United States in the Air, is exact. Can > relevancy be improved? > > It's been quite frustrating to fix some of these issues and then > have them re-appear later because we changed something else that > seemed unrelated. So now I have a cucumber feature like this: > > Feature: Find Relevant Items > In order to find relevant items > As a user > I want to see my expected items on the first screen > > Scenario: united states in the air > Given a blank search > When I search for "united states in the air" > Then I should see record "u4823821" first > > This makes a lot more sense to me than unit tests. I'm not saying > unit tests aren't important, of course they are. But being able to > test the application as a whole seems like something I've been > missing, and I think it's going to help me respond to users and more > easily keep our application behaving the way they expect. > > Bess > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rag9b at eservices.virginia.edu Fri Apr 3 14:43:59 2009 From: rag9b at eservices.virginia.edu (Grizzle, Ronda (rag9b)) Date: Fri, 3 Apr 2009 14:43:59 -0400 Subject: [Blacklight-development] blacklightopac blog update Message-ID: <6918511588AE2E43B10198B8F2DE075A0DC81CD24C@MCCLANE.eservices.virginia.edu> All, I've changed a few things in the template for blacklightopac blog (http://blacklightopac.org/) in response to requests from Naomi and Bess. If you have the chance, please do take a look and let me know of any tweaks that you would like to suggest. Please note that at present the Blog Navigation section of the sidebar is slightly out of whack, but I'm working on it in my very slow, non-php- programmer-editing-php-files sort of way. Thanks, Ronda -- Ronda Grizzle Digital Collections Specialist Digital Collections Services University of Virginia Library -------------- next part -------------- An HTML attachment was scrubbed... URL: From goodieboy at gmail.com Fri Apr 3 15:12:01 2009 From: goodieboy at gmail.com (Matt Mitchell) Date: Fri, 3 Apr 2009 15:12:01 -0400 Subject: [Blacklight-development] RSolr update - 0.8.3 Message-ID: Just wanted to let you all know that I updated RSolr today. It's now @ version 0.8.3. Nothing has changed that will affect Blacklight, so no worries there. I also updated the main README file: http://github.com/mwmitchell/rsolr The main change was adding a #adapter_response method to all of the objects that get returned from solr. For example: response = rsolr.send_request('/document', :q=>'*:*') "response" will have access to the original response through the #adapter_response method: response.adapter_response[:status_code] response.adapter_response[:body] Also, I know some of you have been thinking about sending requests to custom request handlers using RSolr. If you want to use the path variation, use: rsolr.send_request('/document', :q=>1) Instead of the #select method. The #select method uses a preset path of '/select' for convenience. I'm working on re-doing the tests as well. Currently, the tests are using Test::Unit and are some tests require a live Solr connection. I'm changing the tests to either Shoulda or RSpec (probably RSpec) and will also be using mock objects. If anyone wanted to help with the tests, and/or make suggestions on how you'd like the tests to be arranged I wouldn't complain :) Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamie at dang.com Sat Apr 4 00:51:38 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Sat, 4 Apr 2009 00:51:38 -0400 Subject: [Blacklight-development] RSolr update - 0.8.3 In-Reply-To: References: Message-ID: Matt, have a look at Micronaut as well. It is API compatible with RSpec. If you decide to use Cucumber, I believe you'll need RSpec. Jamie On Apr 3, 2009, at 3:12 PM, Matt Mitchell wrote: > Just wanted to let you all know that I updated RSolr today. It's now > @ version 0.8.3. Nothing has changed that will affect Blacklight, so > no worries there. I also updated the main README file: > > http://github.com/mwmitchell/rsolr > > The main change was adding a #adapter_response method to all of the > objects that get returned from solr. For example: > > response = rsolr.send_request('/document', :q=>'*:*') > > "response" will have access to the original response through the > #adapter_response method: > > response.adapter_response[:status_code] > response.adapter_response[:body] > > Also, I know some of you have been thinking about sending requests > to custom request handlers using RSolr. If you want to use the path > variation, use: > > rsolr.send_request('/document', :q=>1) > > Instead of the #select method. The #select method uses a preset path > of '/select' for convenience. > > I'm working on re-doing the tests as well. Currently, the tests are > using Test::Unit and are some tests require a live Solr connection. > I'm changing the tests to either Shoulda or RSpec (probably RSpec) > and will also be using mock objects. > > If anyone wanted to help with the tests, and/or make suggestions on > how you'd like the tests to be arranged I wouldn't complain :) > > Matt > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From erikhatcher at mac.com Sat Apr 4 09:18:12 2009 From: erikhatcher at mac.com (Erik Hatcher) Date: Sat, 04 Apr 2009 09:18:12 -0400 Subject: [Blacklight-development] Fwd: [Blacklight-commits] [1119] trunk/rails/config/initializers/blacklight.rb.example: Updated with current demo app's configuration for search fields. References: <20090403214121.56CBD18580F6@rubyforge.org> Message-ID: <31F16323-D418-48F5-8958-867E2F597194@mac.com> All these .example files lend themselves to a generator. It'd be cool to say something "install blacklight" and then do a script/generate blacklight-config" or something like that, and all these example files are ERb templated and generate their final versions just like that. Thoughts? Erik Begin forwarded message: > From: nobody at rubyforge.org > Date: April 3, 2009 5:41:21 PM EDT > To: blacklight-commits at rubyforge.org > Subject: [Blacklight-commits] [1119] trunk/rails/config/initializers/ > blacklight.rb.example: Updated with current demo app's configuration > for search fields. > > Revision > 1119 > Author > jamieorc > Date > 2009-04-03 17:41:20 -0400 (Fri, 03 Apr 2009) > Log Message > > Updated with current demo app's configuration for search fields. > Modified Paths > > trunk/rails/config/initializers/blacklight.rb.example > Diff > > Modified: trunk/rails/config/initializers/blacklight.rb.example > (1118 => 1119) > --- trunk/rails/config/initializers/blacklight.rb.example 2009-04-03 > 21:41:19 UTC (rev 1118) > +++ trunk/rails/config/initializers/blacklight.rb.example 2009-04-03 > 21:41:20 UTC (rev 1119) > @@ -1 +1,22 @@ > -Blacklight.config[:display_fields][:index_view] > \ No newline at end of file > +# You can configure Blacklight from here. > +# > +# Blacklight.configure(:environment) do |config| end > +# > +# :shared (or leave it blank) is used by all environments. > +# You can override a shared key by using that key in a particular > +# environment's configuration. > +# > +# If you have no configuration beyond :shared for an environment, you > +# do not need to call configure() for that envirnoment. > + > + > +Blacklight.configure(:shared) do |config| > + > + # label is key, solr field is value > + config[:search_fields] ||= [] > + config[:search_fields] << ['All Fields', ''] > + config[:search_fields] << ['Author', 'author_t'] > + config[:search_fields] << ['Title', 'title_t'] > + config[:search_fields] << ['Subject', 'subject_t'] > + > +end > _______________________________________________ > Blacklight-commits mailing list > Blacklight-commits at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-commits -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamie at dang.com Sat Apr 4 09:38:29 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Sat, 4 Apr 2009 09:38:29 -0400 Subject: [Blacklight-development] Fwd: [Blacklight-commits] [1119] trunk/rails/config/initializers/blacklight.rb.example: Updated with current demo app's configuration for search fields. In-Reply-To: <31F16323-D418-48F5-8958-867E2F597194@mac.com> References: <20090403214121.56CBD18580F6@rubyforge.org> <31F16323-D418-48F5-8958-867E2F597194@mac.com> Message-ID: <742905DF-2414-4F0C-B392-F4C21E4DC0E0@dang.com> Could be a useful way to approach it. We had a big discussion about what belongs where yesterday (ie, in the plugin vs in the app), so they'll be some changes coming along WRT config. Moving the non- connectivity stuff out of solr.yml is the first one. On Apr 4, 2009, at 9:18 AM, Erik Hatcher wrote: > All these .example files lend themselves to a generator. It'd be > cool to say something "install blacklight" and then do a script/ > generate blacklight-config" or something like that, and all these > example files are ERb templated and generate their final versions > just like that. > > Thoughts? > > Erik > > > Begin forwarded message: > >> From: nobody at rubyforge.org >> Date: April 3, 2009 5:41:21 PM EDT >> To: blacklight-commits at rubyforge.org >> Subject: [Blacklight-commits] [1119] trunk/rails/config/ >> initializers/blacklight.rb.example: Updated with current demo app's >> configuration for search fields. >> >> Revision >> 1119 >> Author >> jamieorc >> Date >> 2009-04-03 17:41:20 -0400 (Fri, 03 Apr 2009) >> Log Message >> >> Updated with current demo app's configuration for search fields. >> Modified Paths >> >> trunk/rails/config/initializers/blacklight.rb.example >> Diff >> >> Modified: trunk/rails/config/initializers/blacklight.rb.example >> (1118 => 1119) >> --- trunk/rails/config/initializers/blacklight.rb.example >> 2009-04-03 21:41:19 UTC (rev 1118) >> +++ trunk/rails/config/initializers/blacklight.rb.example >> 2009-04-03 21:41:20 UTC (rev 1119) >> @@ -1 +1,22 @@ >> -Blacklight.config[:display_fields][:index_view] >> \ No newline at end of file >> +# You can configure Blacklight from here. >> +# >> +# Blacklight.configure(:environment) do |config| end >> +# >> +# :shared (or leave it blank) is used by all environments. >> +# You can override a shared key by using that key in a particular >> +# environment's configuration. >> +# >> +# If you have no configuration beyond :shared for an environment, >> you >> +# do not need to call configure() for that envirnoment. >> + >> + >> +Blacklight.configure(:shared) do |config| >> + >> + # label is key, solr field is value >> + config[:search_fields] ||= [] >> + config[:search_fields] << ['All Fields', ''] >> + config[:search_fields] << ['Author', 'author_t'] >> + config[:search_fields] << ['Title', 'title_t'] >> + config[:search_fields] << ['Subject', 'subject_t'] >> + >> +end >> _______________________________________________ >> Blacklight-commits mailing list >> Blacklight-commits at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-commits > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From erikhatcher at mac.com Sat Apr 4 09:14:17 2009 From: erikhatcher at mac.com (Erik Hatcher) Date: Sat, 04 Apr 2009 09:14:17 -0400 Subject: [Blacklight-development] demo questions Message-ID: <7A482D06-A4F8-43B5-BCB0-088F576A0A46@mac.com> I'm giving the latest Blacklight a real-world test finally, instead of just speaking from the sidelines. I posted these to IRC yesterday, but then went offline. My use case is this... take a CSV file of books (Mac Delicious Library export, CSV contains ISBN, title, categories, price, and lots more metadata) and import it into Solr and have Blacklight be the UI. Here's the feedback/questions I have: [1:15pm]erikhatcher:http://localhost:3000/catalog/2008308478?counter=1 - what's counter for? [1:15pm]jamieorc:paging [1:16pm]erikhatcher:through what? [1:16pm]jamieorc:the result set [1:23pm]erikhatcher:so the counter (specified on the URL) pages through the result set (ultimately defined in session scope?) So, seems awkward to have some information in session and some on the request. [1:16pm]erikhatcher:next, GETing that page causes two requests to Solr: [1:16pm]erikhatcher:Apr 3, 2009 1:15:50 PM org.apache.solr.core.SolrCore execute [1:16pm]erikhatcher:INFO: [] webapp=/solr path=/select params={id=2008308478&qt=document&wt=ruby} hits=1 status=0 QTime=1 [1:16pm]erikhatcher:Apr 3, 2009 1:15:50 PM org.apache.solr.core.SolrCore execute [1:16pm]erikhatcher:INFO: [] webapp=/solr path=/select params = {wt = ruby &rows = 1 &start =1&facet=false&fq=geographic_subject_facet:"China"&fl=*&qt=search} hits=1 status=0 QTime=1 [1:16pm]erikhatcher:what's with that second request for 1 row? Ok, so why does one request to a Blacklight detail page result in two requests to Solr? That can't be necessary, or right, no? [1:29pm]erikhatcher:just a quick update on where i'm at... i've gotten the blacklight demo to work, just following the wiki instructions (nice work!) [1:29pm]erikhatcher:and then deleted the solr index, restarted, and indexed my CSV of books into solr using this: mapping = { :id => :'amazon link', # that's the column key in the CSV file :datasource_facet => csv_filename, :category_facet => lambda {|data| data[:genres].split('/').map {| s| s.strip}}, :title_t => :title, } indexer = Solr::Indexer.new(source, mapping, :debug => debug) indexer.index do |record, solr_document| # can modify solr_document before it is indexed here end But then I hit this - [1:29pm]erikhatcher:then adjusted solr.yml a little, and got facets and titles showing [1:30pm]erikhatcher:oh wait, facets aren't showing right ... hmmm... investigating.... [1:31pm]erikhatcher:so i adjusted solr.yml to have category_facet only [1:31pm]erikhatcher:under facet_fields [1:31pm]erikhatcher:yes? [1:32pm]erikhatcher:but i still see the four original demo facet fields [1:32pm]erikhatcher:why is that? [1:32pm]erikhatcher:ah, solrconfig.xml! [1:33pm]erikhatcher:so why not use solr.yml's facet_fields to control the &facet.field params to solr? [1:34pm]erikhatcher:i really think having a domain(/site) specific object model (or plural) that is used to generate solrconfig.xml and schema.xml (or fractions thereof) is the way to go here [1:34pm]erikhatcher:as it is, i have to adjust solr.yml (not sure what that changes on the ui yet) AND solrconfig.xml to get facets shown using my "category_facet" rather than "subject_facet" At the very least, solr.yml/schema.xml/solrconfig.xml are a group that needs to stay in sync, so the instructions on the wiki to copy solr.example (let's rename to solr.yml.example to keep the extension there) -> solr.yml should also have solrconfig.xml.example and such to go along with it, eh? And finally success: [1:43pm]erikhatcher:ok, so adjusted solrconfig's facet.field settings, and now browsing by my own category field... sweet [1:44pm]erikhatcher:so overall, an A for Blacklight up and runnability! [1:44pm]erikhatcher:could be a bit easier, but not too shabby Attached is a screenshot of where I got with only a few minutes of fiddling with Blacklight. I'm smiling! But there's more I plan, so it'll be interesting to see how the current Blacklight can handle these: * Multi-select faceting * Ability not to clear facets when typing in a new query * Show book cover images (I have an Amazon ID and ISBN, how to leverage that for images with built-in helpers?) Of course I also want to override the CSS and look a bit, as well as customize the hit and detail display - those will be trivial, I'm sure. Looking forward to feedback on the above questions/issues. Thanks, Erik p.s. My motivation for playing with Blacklight deeper is to write a case study about it to Lucene in Action 2 (or to add a bit about it the Solr chapter I'm now writing). I'll run a draft of whatever I write about Blacklight to Bess and Naomi, and they can have a chance to edit and suggest changes. -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedGraphic.jpg Type: image/jpeg Size: 176429 bytes Desc: not available URL: -------------- next part -------------- From goodieboy at gmail.com Sat Apr 4 12:01:25 2009 From: goodieboy at gmail.com (Matt Mitchell) Date: Sat, 4 Apr 2009 12:01:25 -0400 Subject: [Blacklight-development] demo questions In-Reply-To: <7A482D06-A4F8-43B5-BCB0-088F576A0A46@mac.com> References: <7A482D06-A4F8-43B5-BCB0-088F576A0A46@mac.com> Message-ID: Glad you're checking it out Erik! On Sat, Apr 4, 2009 at 9:14 AM, Erik Hatcher wrote: > I'm giving the latest Blacklight a real-world test finally, instead of just > speaking from the sidelines. I posted these to IRC yesterday, but then went > offline. > > My use case is this... take a CSV file of books (Mac Delicious Library > export, CSV contains ISBN, title, categories, price, and lots more metadata) > and import it into Solr and have Blacklight be the UI. > > Here's the feedback/questions I have: > > [1:15pm]erikhatcher:http://localhost:3000/catalog/2008308478?counter=1 - > what's counter for? > [1:15pm]jamieorc:paging > [1:16pm]erikhatcher:through what? > [1:16pm]jamieorc:the result set > [1:23pm]erikhatcher:so the counter (specified on the URL) pages through the > result set (ultimately defined in session scope?) > > So, seems awkward to have some information in session and some on the > request. There are two things happening here. We keep the search params in the session so we can do next/prev, in the context of the search constraints. The counter is used to know where we're at in the list of matching records. We're using the session over the url for search params to handle the privacy issue. I'm not entirely clear as to why the counter is in the url, but I think there was a good reason or it. > > > [1:16pm]erikhatcher:next, GETing that page causes two requests to Solr: > [1:16pm]erikhatcher:Apr 3, 2009 1:15:50 PM org.apache.solr.core.SolrCore > execute > [1:16pm]erikhatcher:INFO: [] webapp=/solr path=/select > params={id=2008308478&qt=document&wt=ruby} hits=1 status=0 QTime=1 > [1:16pm]erikhatcher:Apr 3, 2009 1:15:50 PM org.apache.solr.core.SolrCore > execute > [1:16pm]erikhatcher:INFO: [] webapp=/solr path=/select > params={wt=ruby&rows=1&start=1&facet=false&fq=geographic_subject_facet:"China"&fl=*&qt=search} > hits=1 status=0 QTime=1 > [1:16pm]erikhatcher:what's with that second request for 1 row? > > Ok, so why does one request to a Blacklight detail page result in two > requests to Solr? That can't be necessary, or right, no? Yeah, to figure out what is next and previous. Might be a better way to do this? > > [1:29pm]erikhatcher:just a quick update on where i'm at... i've gotten the > blacklight demo to work, just following the wiki instructions (nice work!) > [1:29pm]erikhatcher:and then deleted the solr index, restarted, and indexed > my CSV of books into solr using this: > > mapping = { > :id => :'amazon link', # that's the column key in the CSV file > :datasource_facet => csv_filename, > :category_facet => lambda {|data| data[:genres].split('/').map {|s| > s.strip}}, > :title_t => :title, > } > > indexer = Solr::Indexer.new(source, mapping, :debug => debug) > indexer.index do |record, solr_document| > # can modify solr_document before it is indexed here > end > > > But then I hit this - > > [1:29pm]erikhatcher:then adjusted solr.yml a little, and got facets and > titles showing > [1:30pm]erikhatcher:oh wait, facets aren't showing right ... hmmm... > investigating.... > [1:31pm]erikhatcher:so i adjusted solr.yml to have category_facet only > [1:31pm]erikhatcher:under facet_fields > [1:31pm]erikhatcher:yes? > [1:32pm]erikhatcher:but i still see the four original demo facet fields > [1:32pm]erikhatcher:why is that? > [1:32pm]erikhatcher:ah, solrconfig.xml! > [1:33pm]erikhatcher:so why not use solr.yml's facet_fields to control the > &facet.field params to solr? > [1:34pm]erikhatcher:i really think having a domain(/site) specific object > model (or plural) that is used to generate solrconfig.xml and schema.xml (or > fractions thereof) is the way to go here > [1:34pm]erikhatcher:as it is, i have to adjust solr.yml (not sure what that > changes on the ui yet) AND solrconfig.xml to get facets shown using my > "category_facet" rather than "subject_facet" > > At the very least, solr.yml/schema.xml/solrconfig.xml are a group that > needs to stay in sync, so the instructions on the wiki to copy solr.example > (let's rename to solr.yml.example to keep the extension there) -> solr.yml > should also have solrconfig.xml.example and such to go along with it, eh? My original thought was that we'd try to keep as much in solrconfig.xml as possible using the fl param. But then we have UI mapping issues too. I do like the idea of a centralized config/schema generator, in code/dsl. > > And finally success: > [1:43pm]erikhatcher:ok, so adjusted solrconfig's facet.field settings, and > now browsing by my own category field... sweet > [1:44pm]erikhatcher:so overall, an A for Blacklight up and runnability! > [1:44pm]erikhatcher:could be a bit easier, but not too shabby > > Attached is a screenshot of where I got with only a few minutes of fiddling > with Blacklight. I'm smiling! Sweet! > > > But there's more I plan, so it'll be interesting to see how the current > Blacklight can handle these: > > * Multi-select faceting Hmm, you mean multi-valued facet constraints? The url params can handle this, but I have a feeling that you're talking about something else? > > > * Ability not to clear facets when typing in a new query Yep it can do that. > > * Show book cover images (I have an Amazon ID and ISBN, how to leverage > that for images with built-in helpers?) > We have lots of code to do this, but it's not implemented yet. I built a jquery plugin to do google books, and was hoping to add more source providers too. Have any ideas on how we might implmement this? > Of course I also want to override the CSS and look a bit, as well as > customize the hit and detail display - those will be trivial, I'm sure. > Yep, very easy. Just override in your host app! > > Looking forward to feedback on the above questions/issues. > Great! Looking forward to your thoughts and sugestions! Matt > > Thanks, > Erik > > p.s. My motivation for playing with Blacklight deeper is to write a case > study about it to Lucene in Action 2 (or to add a bit about it the Solr > chapter I'm now writing). I'll run a draft of whatever I write about > Blacklight to Bess and Naomi, and they can have a chance to edit and suggest > changes. > > > > > > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndushay at stanford.edu Sat Apr 4 13:13:15 2009 From: ndushay at stanford.edu (Naomi Dushay) Date: Sat, 4 Apr 2009 10:13:15 -0700 Subject: [Blacklight-development] demo questions In-Reply-To: References: <7A482D06-A4F8-43B5-BCB0-088F576A0A46@mac.com> Message-ID: <086E27C7-5462-4386-AB96-D6F8BB8F8B9F@stanford.edu> Some comments on the solrconfig / solr.yml stuff below On Apr 4, 2009, at 9:01 AM, Matt Mitchell wrote: > Glad you're checking it out Erik! > > On Sat, Apr 4, 2009 at 9:14 AM, Erik Hatcher > wrote: > But then I hit this - > > [1:29pm]erikhatcher:then adjusted solr.yml a little, and got facets > and titles showing > [1:30pm]erikhatcher:oh wait, facets aren't showing right ... hmmm... > investigating.... > [1:31pm]erikhatcher:so i adjusted solr.yml to have category_facet only > [1:31pm]erikhatcher:under facet_fields > [1:31pm]erikhatcher:yes? > [1:32pm]erikhatcher:but i still see the four original demo facet > fields > [1:32pm]erikhatcher:why is that? > [1:32pm]erikhatcher:ah, solrconfig.xml! > [1:33pm]erikhatcher:so why not use solr.yml's facet_fields to > control the &facet.field params to solr? > [1:34pm]erikhatcher:i really think having a domain(/site) specific > object model (or plural) that is used to generate solrconfig.xml and > schema.xml (or fractions thereof) is the way to go here > [1:34pm]erikhatcher:as it is, i have to adjust solr.yml (not sure > what that changes on the ui yet) AND solrconfig.xml to get facets > shown using my "category_facet" rather than "subject_facet" > > At the very least, solr.yml/schema.xml/solrconfig.xml are a group > that needs to stay in sync, so the instructions on the wiki to copy > solr.example (let's rename to solr.yml.example to keep the extension > there) -> solr.yml should also have solrconfig.xml.example and such > to go along with it, eh? > > My original thought was that we'd try to keep as much in > solrconfig.xml as possible using the fl param. But then we have UI > mapping issues too. I do like the idea of a centralized config/ > schema generator, in code/dsl. Hmm ... I disagree. To some extent, I am creating my solr index first, and getting blacklight to use it second. When we think of using blacklight, we think "hey - can we stick it on top of our SOLR index for our Fedora app?" "can we stick it on top of our XXX collection solr index?" I don't want to generate my solr index configuration from the context of blacklight; I want the flexibility to go in either direction: from "oh hey - we want to use blacklight for this, so we need a solr index ..." OR "oh hey - this is a solr index, so let's use blacklight for the UI." Ultimately, you probably have to hand configure a bunch of stuff in solrconfig anyway: your boosting schemes for the dismax handler (qf, pf ...) seem unlikely candidates for autoconfig, as well as deciding which request handlers you'd like. ("everything" "title" "author" "subject" "isbn/issn" "periodical" "whatever" "etc") I agree that we need to have better documentation about how to hook up blacklight to a particular solr instance; what blacklight stuff maps to what in the schema/solrconfig files, and exemplars that demonstrate it. I also agree that I perverted the solr.yml file and we need to get away from that approach. We had a great discussion about that on here, and refactoring is going to happen soon. - Naomi -------------- next part -------------- An HTML attachment was scrubbed... URL: From erikhatcher at mac.com Sun Apr 5 06:54:04 2009 From: erikhatcher at mac.com (Erik Hatcher) Date: Sun, 05 Apr 2009 06:54:04 -0400 Subject: [Blacklight-development] demo questions In-Reply-To: References: <7A482D06-A4F8-43B5-BCB0-088F576A0A46@mac.com> Message-ID: <5BBDE6AE-C1F6-4997-A77E-351F7F14E634@mac.com> On Apr 4, 2009, at 12:01 PM, Matt Mitchell wrote: > [1:16pm]erikhatcher:next, GETing that page causes two requests to > Solr: > [1:16pm]erikhatcher:Apr 3, 2009 1:15:50 PM > org.apache.solr.core.SolrCore execute > [1:16pm]erikhatcher:INFO: [] webapp=/solr path=/select > params={id=2008308478&qt=document&wt=ruby} hits=1 status=0 QTime=1 > [1:16pm]erikhatcher:Apr 3, 2009 1:15:50 PM > org.apache.solr.core.SolrCore execute > [1:16pm]erikhatcher:INFO: [] webapp=/solr path=/select > params > = > {wt > = > ruby > &rows > = > 1 > &start > =1&facet=false&fq=geographic_subject_facet:"China"&fl=*&qt=search} > hits=1 status=0 QTime=1 > [1:16pm]erikhatcher:what's with that second request for 1 row? > > Ok, so why does one request to a Blacklight detail page result in > two requests to Solr? That can't be necessary, or right, no? > > Yeah, to figure out what is next and previous. Might be a better way > to do this? I'll have to ponder on that. It doesn't seem right to have the counter on the URL or to make two requests to Solr to show a detail record. Exactly what information is needed for previous/next? > But there's more I plan, so it'll be interesting to see how the > current Blacklight can handle these: > > * Multi-select faceting > > Hmm, you mean multi-valued facet constraints? The url params can > handle this, but I have a feeling that you're talking about > something else? I'm talking about the checkbox faceting as seen here - http://www.lucidimagination.com/search What this changes in the Solr request, as an example of someone checking both History and Science: * facet.field={!ex=ArbitraryKey}subject_facet * fq={!tag=ArbitraryKey}subject_facet:(History OR Science) So rather than multiple facet choices making it into separate fq parameters, all facet selections within a "group" (within a single facet field in the common case) get OR'd together in a single fq, and the facet.field (or facet.query or facet.date) gets that filter excluded from its counts. I'm curious how much work would be involved in supporting that type of Solr parameter fiddling within Blacklight, but of course in such a way that made it work the current way and this way. > * Ability not to clear facets when typing in a new query > > Yep it can do that. How? :) > * Show book cover images (I have an Amazon ID and ISBN, how to > leverage that for images with built-in helpers?) > > We have lots of code to do this, but it's not implemented yet. I > built a jquery plugin to do google books, and was hoping to add more > source providers too. Have any ideas on how we might implmement this? I don't have any ideas on this other than just a common helper like book_cover_image(id) that then renders whatever magic needed. In my use case, I can just override the hit and detail templates and put in a tag pointing to Amazon book covers. Erik From erikhatcher at mac.com Sun Apr 5 07:11:27 2009 From: erikhatcher at mac.com (Erik Hatcher) Date: Sun, 05 Apr 2009 07:11:27 -0400 Subject: [Blacklight-development] demo questions In-Reply-To: <086E27C7-5462-4386-AB96-D6F8BB8F8B9F@stanford.edu> References: <7A482D06-A4F8-43B5-BCB0-088F576A0A46@mac.com> <086E27C7-5462-4386-AB96-D6F8BB8F8B9F@stanford.edu> Message-ID: On Apr 4, 2009, at 1:13 PM, Naomi Dushay wrote: >> My original thought was that we'd try to keep as much in >> solrconfig.xml as possible using the fl param. But then we have UI >> mapping issues too. I do like the idea of a centralized config/ >> schema generator, in code/dsl. > > Hmm ... I disagree. To some extent, I am creating my solr index > first, and getting blacklight to use it second. When we think of > using blacklight, we think "hey - can we stick it on top of our SOLR > index for our Fedora app?" "can we stick it on top of our XXX > collection solr index?" I don't want to generate my solr index > configuration from the context of blacklight; I want the > flexibility to go in either direction: from "oh hey - we want to > use blacklight for this, so we need a solr index ..." OR "oh hey - > this is a solr index, so let's use blacklight for the UI." No matter what, something in Blacklight needs to know about your schema and request handlers. I see the generator approach as a way to customize a Blacklight very rapidly and easily in the case where it's the driver of the view entirely, but in the cases where it's a secondary add-on one could just add the necessary pieces manually like you're already doing in solr.yml. Also, a generator is really a one- time setup step (one-time per catalog view, perhaps, or other granularity), and any artifacts it produces would be designed for further manual customization. It is, of course, asking Blacklight to be more general purpose and configurable to be able to point at arbitrary Solr's and do the right thing. > Ultimately, you probably have to hand configure a bunch of stuff in > solrconfig anyway: your boosting schemes for the dismax handler > (qf, pf ...) seem unlikely candidates for autoconfig, as well as > deciding which request handlers you'd like. ("everything" "title" > "author" "subject" "isbn/issn" "periodical" "whatever" "etc") There's going to be a bit of stuff that ends up hardcoded into solrconfig, for sure. But as you're seeing, you want more dynamic control from the front-end on qf/pf kinda things. It still remains to be seen whether all these various settings are best configured as different request handlers or a common request handler parameterized by the UI. It may depend on what other purposes your Solr server is serving. > I also agree that I perverted the solr.yml file and we need to get > away from that approach. We had a great discussion about that on > here, and refactoring is going to happen soon. Can someone summarize the discussion and the direction this refactoring is headed? Erik From jamie at dang.com Mon Apr 6 10:30:35 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Mon, 6 Apr 2009 10:30:35 -0400 Subject: [Blacklight-development] demo questions In-Reply-To: <5BBDE6AE-C1F6-4997-A77E-351F7F14E634@mac.com> References: <7A482D06-A4F8-43B5-BCB0-088F576A0A46@mac.com> <5BBDE6AE-C1F6-4997-A77E-351F7F14E634@mac.com> Message-ID: <96199617-AB57-4D2B-9E2C-663C5B191D7F@dang.com> I'll have a look at the double requests this morning. I can't imagine why we're doing it. On Apr 5, 2009, at 6:54 AM, Erik Hatcher wrote: > On Apr 4, 2009, at 12:01 PM, Matt Mitchell wrote: >> [1:16pm]erikhatcher:next, GETing that page causes two requests to >> Solr: >> [1:16pm]erikhatcher:Apr 3, 2009 1:15:50 PM >> org.apache.solr.core.SolrCore execute >> [1:16pm]erikhatcher:INFO: [] webapp=/solr path=/select >> params={id=2008308478&qt=document&wt=ruby} hits=1 status=0 QTime=1 >> [1:16pm]erikhatcher:Apr 3, 2009 1:15:50 PM >> org.apache.solr.core.SolrCore execute >> [1:16pm]erikhatcher:INFO: [] webapp=/solr path=/select >> params >> = >> {wt >> = >> ruby >> &rows >> = >> 1 >> &start >> =1&facet=false&fq=geographic_subject_facet:"China"&fl=*&qt=search} >> hits=1 status=0 QTime=1 >> [1:16pm]erikhatcher:what's with that second request for 1 row? >> >> Ok, so why does one request to a Blacklight detail page result in >> two requests to Solr? That can't be necessary, or right, no? >> >> Yeah, to figure out what is next and previous. Might be a better >> way to do this? > > I'll have to ponder on that. It doesn't seem right to have the > counter on the URL or to make two requests to Solr to show a detail > record. Exactly what information is needed for previous/next? From jamie at dang.com Mon Apr 6 10:52:02 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Mon, 6 Apr 2009 10:52:02 -0400 Subject: [Blacklight-development] demo questions In-Reply-To: <5BBDE6AE-C1F6-4997-A77E-351F7F14E634@mac.com> References: <7A482D06-A4F8-43B5-BCB0-088F576A0A46@mac.com> <5BBDE6AE-C1F6-4997-A77E-351F7F14E634@mac.com> Message-ID: <306F5806-8B8A-40DF-B2FB-B7CFD24DDBE8@dang.com> Here's what's going on: One request is for the document. The other(s) (there can be two more) are for the previous and next docs. Their ids are retrieved and the URLs with their respective ids included are generated. On Apr 5, 2009, at 6:54 AM, Erik Hatcher wrote: > > On Apr 4, 2009, at 12:01 PM, Matt Mitchell wrote: >> [1:16pm]erikhatcher:next, GETing that page causes two requests to >> Solr: >> [1:16pm]erikhatcher:Apr 3, 2009 1:15:50 PM >> org.apache.solr.core.SolrCore execute >> [1:16pm]erikhatcher:INFO: [] webapp=/solr path=/select >> params={id=2008308478&qt=document&wt=ruby} hits=1 status=0 QTime=1 >> [1:16pm]erikhatcher:Apr 3, 2009 1:15:50 PM >> org.apache.solr.core.SolrCore execute >> [1:16pm]erikhatcher:INFO: [] webapp=/solr path=/select >> params >> = >> {wt >> = >> ruby >> &rows >> = >> 1 >> &start >> =1&facet=false&fq=geographic_subject_facet:"China"&fl=*&qt=search} >> hits=1 status=0 QTime=1 >> [1:16pm]erikhatcher:what's with that second request for 1 row? >> >> Ok, so why does one request to a Blacklight detail page result in >> two requests to Solr? That can't be necessary, or right, no? >> >> Yeah, to figure out what is next and previous. Might be a better >> way to do this? > > I'll have to ponder on that. It doesn't seem right to have the > counter on the URL or to make two requests to Solr to show a detail > record. Exactly what information is needed for previous/next? From jamie at dang.com Mon Apr 6 13:52:26 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Mon, 6 Apr 2009 13:52:26 -0400 Subject: [Blacklight-development] Moving to Google Groups Message-ID: <1106F23B-5174-4F2B-A40C-C92D4F4C77E4@dang.com> Due to the slowness of delivery of Rubyforge's mailing list, we've moved over to Google Groups. The name is the same, I've added all active members from this list to the new list. You have probably already received an email from the google group. Everyone was added with regular email delivery, so if you want something different, please adjust your membership settings accordingly. If you've been missed, please join! http://groups.google.com/group/blacklight-development Thanks! Jamie From rochkind at jhu.edu Fri Apr 17 10:38:32 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Fri, 17 Apr 2009 10:38:32 -0400 Subject: [Blacklight-development] indexing with jruby In-Reply-To: References: Message-ID: <49E89468.6070000@jhu.edu> I'm confused, I thought you were using a pure-Java indexer in the first place? Matt Mitchell wrote: > As an experiment... I just hacked BL a bit to run the indexer through jruby + RSolr direct. The results were pretty impressive. I set jruby to server mode with Xmx2048m. I started with a clean index and the result was 209 seconds. With standard ruby going through http, I'm getting 1807 seconds. That's 3.4 minutes compared to 30.11 minutes! > > I say "hacked" because jruby wasn't happy with the bcrypt lib (simple_auth plugin) and I also had to setup a direct connection in a temporary way. I'd be interested in finding an alternative to simple_auth, or atleast find a way to get a non-system-extension based bcrypt replacement. > > Also, I think it'd be valuable to allow configuration of the solr connection too, so that it'd be possible for someone to run BL using a direct jruby connection. All we'd have to do is give RSolr a little more info from the solr.yml file. > > Matt > > From jamie at dang.com Fri Apr 17 10:43:44 2009 From: jamie at dang.com (Jamie Orchard-Hays) Date: Fri, 17 Apr 2009 10:43:44 -0400 Subject: [Blacklight-development] indexing with jruby In-Reply-To: References: Message-ID: <10D17D50-B25D-439A-B56A-6749B953B7A9@dang.com> That's cool Matt. I know Jessie wants to get rid of bcrypt dependancies as he is developing on Windows. Hey Jessie, have you tried using Cygwin? I'll bet bcrypt is available there. On Apr 17, 2009, at 10:33 AM, Matt Mitchell wrote: > As an experiment... I just hacked BL a bit to run the indexer > through jruby + RSolr direct. The results were pretty impressive. I > set jruby to server mode with Xmx2048m. I started with a clean index > and the result was 209 seconds. With standard ruby going through > http, I'm getting 1807 seconds. That's 3.4 minutes compared to 30.11 > minutes! > > I say "hacked" because jruby wasn't happy with the bcrypt lib > (simple_auth plugin) and I also had to setup a direct connection in > a temporary way. I'd be interested in finding an alternative to > simple_auth, or atleast find a way to get a non-system-extension > based bcrypt replacement. > > Also, I think it'd be valuable to allow configuration of the solr > connection too, so that it'd be possible for someone to run BL using > a direct jruby connection. All we'd have to do is give RSolr a > little more info from the solr.yml file. > > Matt > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Fri Apr 17 11:03:42 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Fri, 17 Apr 2009 11:03:42 -0400 Subject: [Blacklight-development] indexing with jruby In-Reply-To: References: <49E89468.6070000@jhu.edu> Message-ID: <49E89A4E.7020507@jhu.edu> I want to do whatever UVa is doing. I have no desire to use some weird leftover solution that the main developers aren't actually using, right? So, yeah, if none of the partners are actually using the ruby indexer, why not just remove it from the project? Something not being used by anyone seems likely to get stale and buggy. Matt Mitchell wrote: > Well, UVa is... but the BL demo is still using a ruby based indexer (lib/tasks) -- I'm pretty sure the goal is to replace that with the Java indexer anyway. Not sure entirely? > > Matt > > On Fri, Apr 17, 2009 at 10:38 AM, Jonathan Rochkind > wrote: > I'm confused, I thought you were using a pure-Java indexer in the first place? > > > Matt Mitchell wrote: > As an experiment... I just hacked BL a bit to run the indexer through jruby + RSolr direct. The results were pretty impressive. I set jruby to server mode with Xmx2048m. I started with a clean index and the result was 209 seconds. With standard ruby going through http, I'm getting 1807 seconds. That's 3.4 minutes compared to 30.11 minutes! > > I say "hacked" because jruby wasn't happy with the bcrypt lib (simple_auth plugin) and I also had to setup a direct connection in a temporary way. I'd be interested in finding an alternative to simple_auth, or atleast find a way to get a non-system-extension based bcrypt replacement. > > Also, I think it'd be valuable to allow configuration of the solr connection too, so that it'd be possible for someone to run BL using a direct jruby connection. All we'd have to do is give RSolr a little more info from the solr.yml file. > > Matt > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > From rossfsinger at gmail.com Fri Apr 17 12:06:40 2009 From: rossfsinger at gmail.com (Ross Singer) Date: Fri, 17 Apr 2009 12:06:40 -0400 Subject: [Blacklight-development] indexing with jruby In-Reply-To: References: <49E89468.6070000@jhu.edu> <49E89A4E.7020507@jhu.edu> Message-ID: <23b83f160904170906q273c233dj42bc1cf602d34d2a@mail.gmail.com> Eh, I'm not sure about this. Doesn't SolrMarc seem sort of a heavy dependency for the demo app? I much prefer the idea of a simple install/rake task to just get everything up and running and indexed so somebody can poke around with BL and see how it suits them. If somebody wants to later "graduate" to SolrMarc, so much the better, but I'd like to see the native ruby indexers stay. -Ross. On Fri, Apr 17, 2009 at 11:16 AM, Matt Mitchell wrote: > Jonathan, > > We put the ruby indexer there originally because SolrMarc wasn't quite ready > at the time and the demo needed to get out the door. My jruby experiment was > just for fun so no worries. If you or anyone else would like to add a > SolrMarc indexer to the demo/plugin, go for it! That'd be an awesome > contribution. > > Matt > > On Fri, Apr 17, 2009 at 11:03 AM, Jonathan Rochkind > wrote: >> >> I want to do whatever UVa is doing. I have no desire to use some weird >> leftover solution that the main developers aren't actually using, right? >> ?So, yeah, if none of the partners are actually using the ruby indexer, why >> not just remove it from the project? Something not being used by anyone >> seems likely to get stale and buggy. >> >> Matt Mitchell wrote: >>> >>> Well, UVa is... but the BL demo is still using a ruby based indexer >>> (lib/tasks) -- I'm pretty sure the goal is to replace that with the Java >>> indexer anyway. Not sure entirely? >>> >>> Matt >>> >>> On Fri, Apr 17, 2009 at 10:38 AM, Jonathan Rochkind >>> > wrote: >>> I'm confused, I thought you were using a pure-Java indexer in the first >>> place? >>> >>> >>> Matt Mitchell wrote: >>> As an experiment... I just hacked BL a bit to run the indexer through >>> jruby + RSolr direct. The results were pretty impressive. I set jruby to >>> server mode with Xmx2048m. I started with a clean index and the result was >>> 209 seconds. With standard ruby going through http, I'm getting 1807 >>> seconds. That's 3.4 minutes compared to 30.11 minutes! >>> >>> I say "hacked" because jruby wasn't happy with the bcrypt lib >>> (simple_auth plugin) and I also had to setup a direct connection in a >>> temporary way. I'd be interested in finding an alternative to simple_auth, >>> or atleast find a way to get a non-system-extension based bcrypt >>> replacement. >>> >>> Also, I think it'd be valuable to allow configuration of the solr >>> connection too, so that it'd be possible for someone to run BL using a >>> direct jruby connection. All we'd have to do is give RSolr a little more >>> info from the solr.yml file. >>> >>> Matt >>> >>> >>> _______________________________________________ >>> Blacklight-development mailing list >>> >>> Blacklight-development at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/blacklight-development >>> Blacklightopac Blog http://blacklightopac.org/ >>> >>> >>> >> >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > From jkeck at stanford.edu Fri Apr 17 12:08:36 2009 From: jkeck at stanford.edu (Jessie Keck) Date: Fri, 17 Apr 2009 09:08:36 -0700 Subject: [Blacklight-development] indexing with jruby In-Reply-To: <10D17D50-B25D-439A-B56A-6749B953B7A9@dang.com> References: <10D17D50-B25D-439A-B56A-6749B953B7A9@dang.com> Message-ID: <49E8A984.10308@stanford.edu> I've heard that cygwin will work with bcrypt. The problem is the nmake command that the gem uses on installation to make files. Windows does not have a nmake command (cygwin will provide this command on windows). However; this just adds to the things that a windows user will need to get this running on their machine for a feature that isn't really being leveraged right now (and will most likely be overridden by almost all institutions for their specific authentication methods). I think a simple authentication method out of the box will be a nice feature to show what additional features can be made available to authenticated users. Also, it looks like the indexer doesn't like it either. I have UserAuth code that can be added directly in the blacklight plugin and would require no further dependencies. I think it's a pretty solid authentication method (pretty secure passwords with MD5 Hash and randomly generated SALT strings) Removing the simple_auth and replacing it with Blacklights own out of the box authentication system is in my queue. -Jessie Keck Jamie Orchard-Hays wrote: > That's cool Matt. > > I know Jessie wants to get rid of bcrypt dependancies as he is > developing on Windows. > > Hey Jessie, have you tried using Cygwin? I'll bet bcrypt is available > there. > > > On Apr 17, 2009, at 10:33 AM, Matt Mitchell wrote: > >> As an experiment... I just hacked BL a bit to run the indexer through >> jruby + RSolr direct. The results were pretty impressive. I set jruby >> to server mode with Xmx2048m. I started with a clean index and the >> result was 209 seconds. With standard ruby going through http, I'm >> getting 1807 seconds. That's 3.4 minutes compared to 30.11 minutes! >> >> I say "hacked" because jruby wasn't happy with the bcrypt lib >> (simple_auth plugin) and I also had to setup a direct connection in a >> temporary way. I'd be interested in finding an alternative to >> simple_auth, or atleast find a way to get a non-system-extension >> based bcrypt replacement. >> >> Also, I think it'd be valuable to allow configuration of the solr >> connection too, so that it'd be possible for someone to run BL using >> a direct jruby connection. All we'd have to do is give RSolr a little >> more info from the solr.yml file. >> >> Matt >> _______________________________________________ >> Blacklight-development mailing list >> Blacklight-development at rubyforge.org >> http://rubyforge.org/mailman/listinfo/blacklight-development >> Blacklightopac Blog http://blacklightopac.org/ > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ From rochkind at jhu.edu Fri Apr 17 14:48:56 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Fri, 17 Apr 2009 14:48:56 -0400 Subject: [Blacklight-development] indexing with jruby In-Reply-To: <49E8C748.8030200@virginia.edu> References: <49E89468.6070000@jhu.edu> <49E89A4E.7020507@jhu.edu> <23b83f160904170906q273c233dj42bc1cf602d34d2a@mail.gmail.com> <49E8C748.8030200@virginia.edu> Message-ID: <49E8CF18.1060007@jhu.edu> This is awesome Bob, great work. Robert Haschart wrote: > I have a pre-built binary version of SolrMarc ready to go, that is customized for the Blacklight Demo, it consists of a single jar file that contains all of the necessary java code and properties files, (except for Solr libs) plus a number of shell scripts (or batch files if you are in PC-land) for configuring, tweaking and running SolrMarc. > > The goal is to be able to go from downloading and unpacking the file, to having a usuable index to use with a sample Blacklight instance in about 5 minutes. I think at this point the only prerequisite for running the prebuilt binary version, is the ability to run a java program (ie. an installed JRE of version 1.5 or newer) which you will need in any case to run tomcat or jetty (or solr itself) > > The binary version can be modified and customized using the provided scripts, to change what fields are indexed and how values are translated from the value in the Marc record to a friendlier, user-readable version . It is only when a site's need for customization gets to the point that it needs to write a custom java indexing routine that they will need to upgrade to full source distribution of SolrMarc. > > I believe that this pre-built binary SolrMarc is ready to be included as a part of the Blacklight Demo distribution. If it is placed in a directory named indexer in the distribution next to the jetty directory and the rails directory, all you will need to do is run the following scripts: > > indexer/setsolrwar ./jetty/webapps/solr.war > indexer/setsolrhome ./jetty/solr > indexer/indexfile data/test_data.utf8.mrc > indexer/indexfile data/lc_records.utf8.mrc > > or if you are lazy you could do the last two commands at the same time like this: > > cat data/*.mrc | indexer/indexfile > > -Bob Haschart > > > Ross Singer wrote: > > Eh, I'm not sure about this. Doesn't SolrMarc seem sort of a heavy > dependency for the demo app? I much prefer the idea of a simple > install/rake task to just get everything up and running and indexed so > somebody can poke around with BL and see how it suits them. > > If somebody wants to later "graduate" to SolrMarc, so much the better, > but I'd like to see the native ruby indexers stay. > > -Ross. > > On Fri, Apr 17, 2009 at 11:16 AM, Matt Mitchell wrote: > > > Jonathan, > > We put the ruby indexer there originally because SolrMarc wasn't quite ready > at the time and the demo needed to get out the door. My jruby experiment was > just for fun so no worries. If you or anyone else would like to add a > SolrMarc indexer to the demo/plugin, go for it! That'd be an awesome > contribution. > > Matt > > On Fri, Apr 17, 2009 at 11:03 AM, Jonathan Rochkind > wrote: > > > I want to do whatever UVa is doing. I have no desire to use some weird > leftover solution that the main developers aren't actually using, right? > So, yeah, if none of the partners are actually using the ruby indexer, why > not just remove it from the project? Something not being used by anyone > seems likely to get stale and buggy. > > Matt Mitchell wrote: > > > Well, UVa is... but the BL demo is still using a ruby based indexer > (lib/tasks) -- I'm pretty sure the goal is to replace that with the Java > indexer anyway. Not sure entirely? > > Matt > > On Fri, Apr 17, 2009 at 10:38 AM, Jonathan Rochkind > > wrote: > I'm confused, I thought you were using a pure-Java indexer in the first > place? > > > Matt Mitchell wrote: > As an experiment... I just hacked BL a bit to run the indexer through > jruby + RSolr direct. The results were pretty impressive. I set jruby to > server mode with Xmx2048m. I started with a clean index and the result was > 209 seconds. With standard ruby going through http, I'm getting 1807 > seconds. That's 3.4 minutes compared to 30.11 minutes! > > I say "hacked" because jruby wasn't happy with the bcrypt lib > (simple_auth plugin) and I also had to setup a direct connection in a > temporary way. I'd be interested in finding an alternative to simple_auth, > or atleast find a way to get a non-system-extension based bcrypt > replacement. > > Also, I think it'd be valuable to allow configuration of the solr > connection too, so that it'd be possible for someone to run BL using a > direct jruby connection. All we'd have to do is give RSolr a little more > info from the solr.yml file. > > Matt > > > _______________________________________________ > Blacklight-development mailing list > > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > > _______________________________________________ > Blacklight-development mailing list > Blacklight-development at rubyforge.org > http://rubyforge.org/mailman/listinfo/blacklight-development > Blacklightopac Blog http://blacklightopac.org/ > > > >