From randy.metcalfe at gmail.com  Thu Mar  6 10:55:50 2008
From: randy.metcalfe at gmail.com (Randy Metcalfe)
Date: Thu, 6 Mar 2008 10:55:50 -0500
Subject: [Blacklight-development] svn update problem
Message-ID: <b66c649c0803060755s2c84e3admb801dc56258518e5@mail.gmail.com>

Hi,

It's been a while since I updated my local copy of Blacklight from the
svn repository. Today when I ran svn update I got the following
message (after a fair numbers of updates had already come through).

Fetching external item into 'branches/uva_lib/trunk/rails/vendor/plugins/flare'
mwmitchell at rubyforge.org's password:

Nothing hangs on this since I'm just viewing the code, not developing
it. So this is just a heads up.

Cheers,

Randy

-- 
Randy Metcalfe

From rochkind at jhu.edu  Thu Mar 13 11:31:38 2008
From: rochkind at jhu.edu (Jonathan Rochkind)
Date: Thu, 13 Mar 2008 11:31:38 -0400
Subject: [Blacklight-development] solr schema; facetting vs searching
In-Reply-To: <D509BEF7-8F78-4C0E-BA32-96BB4D8BE7CC@mac.com>
References: <47B07188.9030103@jhu.edu>
	<8D9C52C8-A282-4FF0-AA18-F6CF828C5314@mac.com>
	<47B1BE6A.7060203@jhu.edu> <47B4B377.2040809@jhu.edu>
	<B9AD5F7E-29FA-4B20-A753-E11F8F36D683@mac.com>
	<47B5EF63.9030102@jhu.edu>
	<D509BEF7-8F78-4C0E-BA32-96BB4D8BE7CC@mac.com>
Message-ID: <47D948DA.30305@jhu.edu>

So I finally have luke up and running to look at this stuff!

I expected luke would give me a way to look through all the indexes and 
tell me the characterstics/settings of each one, but note quite.

It does let me look at each document, and how it's indexed. I expected 
the index called "text" would have all searchable words for each 
document? But when I look at a document in luke, 'text' is empty for all 
of them. "<not present or not stored>".

In general, what I want to figure out next is where those "_text" fields 
are coming from. I could probably hack the indexer and/or the solr 
schema so that every single *_facet field automatically got a 
corresponding searchable *_index field, yes? Any hints as to where I'd 
do this? (Not sure if I'd do it in the indexer code, the solr schema 
file, or both).

Jonathan

Erik Hatcher wrote:
>
> On Feb 15, 2008, at 3:00 PM, Jonathan Rochkind wrote:
>> Perfect, thanks. Before I was asking if you could do a fielded search in
>> the current blacklight. You said, yes, using lucene syntax, if you knew
>> the name of the field you wanted to search. But it sounds like most of
>> the fields you might want to search actually aren't stored in the index
>> in a suitable way at present.
>
> I wouldn't say that.   There are a lot of *_text fields that are 
> surely useful for searching.   And take a look at the schema.xml - all 
> fields are being stored currently so they are 
> retrievable/highlightable from the front-end easily.
>
> Have a look at the Lucene index using Luke 
> <http://www.getopt.org/luke/> or Solr's Luke Request Handler - check 
> out the fields indexed and play around.   There is more there than 
> meets the eye.   Certainly there is much more to MARC than the indexer 
> is extracting currently, though - so lots more can be done.
>
>> If Blacklight doesn't do sortable fields yet.... how does Blacklight
>> sort search results?
>
> Depends - if you're purely navigating by facets, the order is in index 
> order (the order the records were indexed).
>
> However, once you add a full-text query, sorting is by.... 
> relevance!   Imagine that :)
>
>     Erik

-- 
Jonathan Rochkind
Digital Services Software Engineer
The Sheridan Libraries
Johns Hopkins University
410.516.8886 
rochkind (at) jhu.edu


From rochkind at jhu.edu  Thu Mar 13 11:35:26 2008
From: rochkind at jhu.edu (Jonathan Rochkind)
Date: Thu, 13 Mar 2008 11:35:26 -0400
Subject: [Blacklight-development] LCSH facets;  international chars
In-Reply-To: <D509BEF7-8F78-4C0E-BA32-96BB4D8BE7CC@mac.com>
References: <47B07188.9030103@jhu.edu>
	<8D9C52C8-A282-4FF0-AA18-F6CF828C5314@mac.com>
	<47B1BE6A.7060203@jhu.edu> <47B4B377.2040809@jhu.edu>
	<B9AD5F7E-29FA-4B20-A753-E11F8F36D683@mac.com>
	<47B5EF63.9030102@jhu.edu>
	<D509BEF7-8F78-4C0E-BA32-96BB4D8BE7CC@mac.com>
Message-ID: <47D949BE.9000507@jhu.edu>

Sorry for more email, today's another of my "explore blacklight and 
SOLR" days.

It looks like you are putting LCSH's into facets without 
"uncoordinating" them. That is, there is a "topic_form_genre" facet (and 
display field) rather than seperating out topic, form, and genre. Yes?  
Any background on this?  (Just didn't get to this; found it 
preferable?).  Also, the Blacklight demo I have doesn't actually expose 
this "topic_form_genre_facet" in the facets pane, it's not available for 
browsing. Again, any background?

Also, I'm seeing some records in the same set that came with blacklight 
that have international characters that are NOT displaying properly. In 
general, do you have proper display of non-ascii chars working?  Record 
in example set that gives me improperly displayed records is title "The 
art of the persian santur".

Jonathan

From erikhatcher at mac.com  Thu Mar 13 21:24:27 2008
From: erikhatcher at mac.com (Erik Hatcher)
Date: Thu, 13 Mar 2008 21:24:27 -0400
Subject: [Blacklight-development] solr schema; facetting vs searching
In-Reply-To: <47D948DA.30305@jhu.edu>
References: <47B07188.9030103@jhu.edu>
	<8D9C52C8-A282-4FF0-AA18-F6CF828C5314@mac.com>
	<47B1BE6A.7060203@jhu.edu> <47B4B377.2040809@jhu.edu>
	<B9AD5F7E-29FA-4B20-A753-E11F8F36D683@mac.com>
	<47B5EF63.9030102@jhu.edu>
	<D509BEF7-8F78-4C0E-BA32-96BB4D8BE7CC@mac.com>
	<47D948DA.30305@jhu.edu>
Message-ID: <3F68DE83-494D-4557-908B-1081655107A6@mac.com>


On Mar 13, 2008, at 11:31 AM, Jonathan Rochkind wrote:
> I expected luke would give me a way to look through all the indexes  
> and tell me the characterstics/settings of each one, but note quite.
>
> It does let me look at each document, and how it's indexed. I  
> expected the index called "text" would have all searchable words  
> for each document? But when I look at a document in luke, 'text' is  
> empty for all of them. "<not present or not stored>".

Just a terminology adjustment here.  In Lucene parlance, "text" is  
considered _field_.  An index is the entire directory created by  
Lucene (Solr's data/index directory).

Empty is not quite true.  The text field is indexed, but not stored.   
This is because it is a purely aggregate field of all other stored  
fields.  In Luke's document tab view, click the "Reconstruct & Edit"  
button, then look at the text field and the inner "Tokenized" tab to  
see the terms that were indexed for that particular field.

Or, from Solr, hit http://localhost:8983/solr/admin/luke?fl=text to  
get stats on that field and top terms, etc.

> In general, what I want to figure out next is where those "_text"  
> fields are coming from.

look at scripts/virgo_marc_map.rb, search for _text in there.  That's  
where the fields come from.

> I could probably hack the indexer and/or the solr schema so that  
> every single *_facet field automatically got a corresponding  
> searchable *_index field, yes?

To be clear, the *_facet fields are _searchable_ they just aren't  
tokenized.  So you can search for format_facet:"Digital Media" as an  
exact term match.  But you wouldn't find it by searching for just  
"digital" or "media" (on that exact field).  *_facet fields are,  
however, copied into "text", which is of course tokenized and  
searchable on individual words.

But you could <copyField source="*_facet" dest="*_text"/> if you like  
to have facet fields also individually searchable by words using the  
*_text version.

> Any hints as to where I'd do this? (Not sure if I'd do it in the  
> indexer code, the solr schema file, or both).

Either would work, but Solr's <copyField> handles this scenario.

	Erik


From erikhatcher at mac.com  Thu Mar 13 21:41:52 2008
From: erikhatcher at mac.com (Erik Hatcher)
Date: Thu, 13 Mar 2008 21:41:52 -0400
Subject: [Blacklight-development] LCSH facets;  international chars
In-Reply-To: <47D949BE.9000507@jhu.edu>
References: <47B07188.9030103@jhu.edu>
	<8D9C52C8-A282-4FF0-AA18-F6CF828C5314@mac.com>
	<47B1BE6A.7060203@jhu.edu> <47B4B377.2040809@jhu.edu>
	<B9AD5F7E-29FA-4B20-A753-E11F8F36D683@mac.com>
	<47B5EF63.9030102@jhu.edu>
	<D509BEF7-8F78-4C0E-BA32-96BB4D8BE7CC@mac.com>
	<47D949BE.9000507@jhu.edu>
Message-ID: <570BA11A-CCF3-4AF6-873F-8A147202820F@mac.com>


On Mar 13, 2008, at 11:35 AM, Jonathan Rochkind wrote:
> Sorry for more email, today's another of my "explore blacklight and
> SOLR" days.

What a glorious day it is!

> It looks like you are putting LCSH's into facets without
> "uncoordinating" them. That is, there is a "topic_form_genre" facet  
> (and
> display field) rather than seperating out topic, form, and genre. Yes?
> Any background on this?  (Just didn't get to this; found it
> preferable?).

I created the initial facets based on conversations with UVa's  
(former) head of cataloging and just kinda winged it from there  
pulling out what looked good to me. Since my time on it, they may  
have adjusted some of these.  In general I think it was somewhat  
thought through, but also it was well known that these mappings were  
up for debate and rearrangement.

It's adjustable in scripts/virgo_marc_map.rb:

   :topic_form_genre_facet => [:'650a', :'650b', :'650x', :'655a']


>   Also, the Blacklight demo I have doesn't actually expose
> this "topic_form_genre_facet" in the facets pane, it's not  
> available for
> browsing. Again, any background?

I exposed all facets in my original incarnation, but since then it  
was deemed too much for the UI, so they scaled it back.  I've been  
chatting with Matt about it, and what I'd like to see is the facets  
are selectable by the user via some preferences.  I WANT THOSE FACETS  
BACK!  :)   I love browsing by the headings and see how intermingle  
with other facets.

> Also, I'm seeing some records in the same set that came with  
> blacklight
> that have international characters that are NOT displaying  
> properly. In
> general, do you have proper display of non-ascii chars working?   
> Record
> in example set that gives me improperly displayed records is title  
> "The
> art of the persian santur".

In general, yes, Blacklight worked fine with international characters  
back in my time.  I'm not sure if something broke that or if maybe  
something else is fishy in your environment that is causing it.  I  
had issues with non-ascii characters in my first indexing runs, but  
then discovered the MARC files themselves were mangled encoding-wise  
and I ran them through some yaz tools magic and that fixed things up  
fine.  Maybe the test MARC file needs the same conversion run on it?   
I just confirmed that I see the mangled stuff myself, and I'm  
guessing the MARC file needs some encoding TLC.  Matt?  Bess?

	Erik


From goodieboy at gmail.com  Fri Mar 14 17:12:16 2008
From: goodieboy at gmail.com (Matt M.)
Date: Fri, 14 Mar 2008 17:12:16 -0400
Subject: [Blacklight-development] Flare and Blacklight - Demos
Message-ID: <c66dcac40803141412q7958a77cx81fd626bb50766ff@mail.gmail.com>

Just wanted to let everyone know that the demos should be runnable/working.
The demos are located here:

http://blacklight.rubyforge.org/svn/demos/

FLARE - faceted browsing/searching using Rails and Solr
For the Flare demo, execute (within the rails directory):

rake flare:demo

This should start-up Solr via Jetty, index an RSS feed from
mystrands.comand start up Rails. Please wait for the indexing to
complete before browsing
to:
http://localhost:3000/documents

BLACKLIGHT - Built on top of Flare, also provides code for working within
library domain (MARC, Z3950 etc.)
For the Blacklight demo, execute (within the rails directory):

rake blacklight:demo

This should start-up Solr via Jetty, index a demo MARC file and start up
Rails. Please wait for the indexing to complete before browsing to:
http://localhost:3000/documents

Please let me know if you have any problems with dependencies/bugs, or
suggestions!

These are just demos, and everything is still in alpha stage...
API/interfaces WILL be changing.

Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/blacklight-development/attachments/20080314/21542772/attachment.html 

From goodieboy at gmail.com  Mon Mar 17 14:33:25 2008
From: goodieboy at gmail.com (Matt Mitchell)
Date: Mon, 17 Mar 2008 14:33:25 -0400
Subject: [Blacklight-development] Flare and Blacklight - Demos
In-Reply-To: <47DE8165.3080508@jhu.edu>
References: <c66dcac40803141412q7958a77cx81fd626bb50766ff@mail.gmail.com>
	<47DE8165.3080508@jhu.edu>
Message-ID: <c66dcac40803171133h78e0894alc1bb51b0c3ec5d62@mail.gmail.com>

Hi Jonathan,

The Flare demo is not using the old routine. It's using a rake task that
queries a live web service for XML. The Blacklight demo is using the old
system, but it's been cleaned up quite a bit. In fact the Blacklight Demo
indexing implementation is now in one file:

http://blacklight.rubyforge.org/svn/rails-plugins/blacklight/trunk/lib/blacklight/demo/uva.rb

I put it in with the Blacklight plugin because it could be a valuable
reference for someone trying to use the plugin from scratch.

Hope this helps!

Matt

On Mon, Mar 17, 2008 at 10:34 AM, Jonathan Rochkind <rochkind at jhu.edu>
wrote:

> Are these using the new indexing routines, or the old ones? Thanks!
>
> Jonathan
>
> Matt M. wrote:
> > Just wanted to let everyone know that the demos should be
> runnable/working.
> > The demos are located here:
> >
> > http://blacklight.rubyforge.org/svn/demos/
> >
> > FLARE - faceted browsing/searching using Rails and Solr
> > For the Flare demo, execute (within the rails directory):
> >
> > rake flare:demo
> >
> > This should start-up Solr via Jetty, index an RSS feed from
> > mystrands.comand start up Rails. Please wait for the indexing to
> > complete before browsing
> > to:
> > http://localhost:3000/documents
> >
> > BLACKLIGHT - Built on top of Flare, also provides code for working
> within
> > library domain (MARC, Z3950 etc.)
> > For the Blacklight demo, execute (within the rails directory):
> >
> > rake blacklight:demo
> >
> > This should start-up Solr via Jetty, index a demo MARC file and start up
> > Rails. Please wait for the indexing to complete before browsing to:
> > http://localhost:3000/documents
> >
> > Please let me know if you have any problems with dependencies/bugs, or
> > suggestions!
> >
> > These are just demos, and everything is still in alpha stage...
> > API/interfaces WILL be changing.
> >
> > Matt
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Blacklight-development mailing list
> > Blacklight-development at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/blacklight-development
> >
>
> --
> Jonathan Rochkind
> Digital Services Software Engineer
> The Sheridan Libraries
> Johns Hopkins University
> 410.516.8886
> rochkind (at) jhu.edu
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/blacklight-development/attachments/20080317/4b29d5f5/attachment-0001.html 

From goodieboy at gmail.com  Mon Mar 17 15:25:39 2008
From: goodieboy at gmail.com (Matt Mitchell)
Date: Mon, 17 Mar 2008 15:25:39 -0400
Subject: [Blacklight-development] Flare and Blacklight - Demos
In-Reply-To: <47DEBBCC.2070907@jhu.edu>
References: <c66dcac40803141412q7958a77cx81fd626bb50766ff@mail.gmail.com>
	<47DE8165.3080508@jhu.edu>
	<c66dcac40803171133h78e0894alc1bb51b0c3ec5d62@mail.gmail.com>
	<47DEBBCC.2070907@jhu.edu>
Message-ID: <c66dcac40803171225o7de020dclf7465328f947c7ac@mail.gmail.com>

Yes, that is coming very soon and will be part of the repo, but probably in
it's own root directory.

Matt

On Mon, Mar 17, 2008 at 2:43 PM, Jonathan Rochkind <rochkind at jhu.edu> wrote:

> Cool. I'm somewhat confused now. I was thinking of the 'new'
> Java-implementation of indexing that I heard was coming. Is that
> available in the blacklight svn yet, and if so, how would I look it at
> and/or demo it?
>
> I agree with you on the usefulness of having indexing demos in the
> blacklight distro.
>
> Jonathan
>
> Matt Mitchell wrote:
> > Hi Jonathan,
> >
> > The Flare demo is not using the old routine. It's using a rake task that
> > queries a live web service for XML. The Blacklight demo is using the old
> > system, but it's been cleaned up quite a bit. In fact the Blacklight
> Demo
> > indexing implementation is now in one file:
> >
> >
> http://blacklight.rubyforge.org/svn/rails-plugins/blacklight/trunk/lib/blacklight/demo/uva.rb
> >
> > I put it in with the Blacklight plugin because it could be a valuable
> > reference for someone trying to use the plugin from scratch.
> >
> > Hope this helps!
> >
> > Matt
> >
> > On Mon, Mar 17, 2008 at 10:34 AM, Jonathan Rochkind <rochkind at jhu.edu>
> > wrote:
> >
> >
> >> Are these using the new indexing routines, or the old ones? Thanks!
> >>
> >> Jonathan
> >>
> >> Matt M. wrote:
> >>
> >>> Just wanted to let everyone know that the demos should be
> >>>
> >> runnable/working.
> >>
> >>> The demos are located here:
> >>>
> >>> http://blacklight.rubyforge.org/svn/demos/
> >>>
> >>> FLARE - faceted browsing/searching using Rails and Solr
> >>> For the Flare demo, execute (within the rails directory):
> >>>
> >>> rake flare:demo
> >>>
> >>> This should start-up Solr via Jetty, index an RSS feed from
> >>> mystrands.comand start up Rails. Please wait for the indexing to
> >>> complete before browsing
> >>> to:
> >>> http://localhost:3000/documents
> >>>
> >>> BLACKLIGHT - Built on top of Flare, also provides code for working
> >>>
> >> within
> >>
> >>> library domain (MARC, Z3950 etc.)
> >>> For the Blacklight demo, execute (within the rails directory):
> >>>
> >>> rake blacklight:demo
> >>>
> >>> This should start-up Solr via Jetty, index a demo MARC file and start
> up
> >>> Rails. Please wait for the indexing to complete before browsing to:
> >>> http://localhost:3000/documents
> >>>
> >>> Please let me know if you have any problems with dependencies/bugs, or
> >>> suggestions!
> >>>
> >>> These are just demos, and everything is still in alpha stage...
> >>> API/interfaces WILL be changing.
> >>>
> >>> Matt
> >>>
> >>>
> >>>
> ------------------------------------------------------------------------
> >>>
> >>> _______________________________________________
> >>> Blacklight-development mailing list
> >>> Blacklight-development at rubyforge.org
> >>> http://rubyforge.org/mailman/listinfo/blacklight-development
> >>>
> >>>
> >> --
> >> Jonathan Rochkind
> >> Digital Services Software Engineer
> >> The Sheridan Libraries
> >> Johns Hopkins University
> >> 410.516.8886
> >> rochkind (at) jhu.edu
> >>
> >>
> >>
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Blacklight-development mailing list
> > Blacklight-development at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/blacklight-development
> >
>
> --
> Jonathan Rochkind
> Digital Services Software Engineer
> The Sheridan Libraries
> Johns Hopkins University
> 410.516.8886
> rochkind (at) jhu.edu
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/blacklight-development/attachments/20080317/5b3b3fc5/attachment.html 

From jamie at dang.com  Mon Mar 24 14:05:01 2008
From: jamie at dang.com (Jamie Orchard-Hays)
Date: Mon, 24 Mar 2008 14:05:01 -0400
Subject: [Blacklight-development] Engines?
Message-ID: <B25E503F-9FE8-49D5-930F-6F3877C0CB60@dang.com>

I'm curious if BL is going to use rails Engines plugin or if that idea  
was abandoned. I see there are no plugins in the source code right now.

Jamie

From goodieboy at gmail.com  Mon Mar 24 15:34:55 2008
From: goodieboy at gmail.com (Matt Mitchell)
Date: Mon, 24 Mar 2008 15:34:55 -0400
Subject: [Blacklight-development] Engines?
In-Reply-To: <B25E503F-9FE8-49D5-930F-6F3877C0CB60@dang.com>
References: <B25E503F-9FE8-49D5-930F-6F3877C0CB60@dang.com>
Message-ID: <c66dcac40803241234q6edf38afw4e5a8404e7797136@mail.gmail.com>

Hi Jamie!

I thought about using Engines. I want to look into that as soon as I get
some more time. But the current Blacklight works in an "Engines" kind of
way. Blacklight and Flare are implemented as plugins actually, you won't see
it in the plugins directory though, because it's brought in by
svn:externals.

The basic application hierarchy is:

uva-blacklight < blacklight < flare

- Flare provides the basic Rails UI and querying building for Solr
- Blacklight uses Flare and contains additional "library" code
- uva-blacklight uses all of the above, but customizes some of the
views/helpers to look fancy. uva-blacklight is also the real Rails
application.

The details...

Flare is a plugin, but it also has the basic app files; controllers, views,
helpers. Flare is "booted" in your application's environment.rb file, within
the config.initialize block. It's during this "boot" process that Flare is
able to add it's load and view paths to the current Rails configuration.
After that happens, the usual init.rb file is loaded by the Rails plugin
framework. So you can install Flare, and have an app up and running in no
time. As soon as you want to override something, you can simply re-create
the parallel file in your application's "app" directory.

The Blacklight plugin is similar. It's a plugin, but it also has app files
that can be "masked" by parallel files in your "app" directory; it's booted
like Flare. The Blacklight plugin uses Flare as a plugin. And because of
that, it's strikingly similar, but it contains additional "library" domain
code (MARC, Z3950 etc.). Flare is linked in to Blacklight via svn:externals.
The Blacklight plugin can mask the Flare files, and the app that uses the
Blacklight plugin (UVa's implementation for example) can mask Blacklights
files.

The Blacklight plugin is a work in progress... as soon as the UVa
implementation solves a particular problem, I try to abstract it into
something more generally useful and then put it into the Blacklight plugin's
lib directory. For example, last week I created a module that loads book
covers from multiple sources (Syndetics, Google and Amazon). I plan on
putting that into Blacklight sometime this week. There will also be more
Z3950 code coming.

The UVa Blacklight uses svn:externals on both Flare and Blacklight.

I'd love to look at Engines again sometime. The main reason I didn't go with
Engines is because I wanted to avoid anything with too much "magic". I think
Engines will automatically mixin code (controllers, models helpers) and some
of that freaks me a bit. But who knows, it could be a very useful thing for
us.

The source for the plugins is here:

http://blacklight.rubyforge.org/svn/rails-plugins/

For a basic usage of Flare and Blacklight, see:

http://blacklight.rubyforge.org/svn/demos/
- you won't see anything in the plugins directory until you do a checkout
- have a look at the config/environment.rb files to see how the booting
works

I'd love to hear your thoughts, concerns and/or ideas.

Matt

On Mon, Mar 24, 2008 at 2:05 PM, Jamie Orchard-Hays <jamie at dang.com> wrote:

> I'm curious if BL is going to use rails Engines plugin or if that idea
> was abandoned. I see there are no plugins in the source code right now.
>
> Jamie
> _______________________________________________
> Blacklight-development mailing list
> Blacklight-development at rubyforge.org
> http://rubyforge.org/mailman/listinfo/blacklight-development
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/blacklight-development/attachments/20080324/88dc90d3/attachment.html