[Blacklight-development] refactoring blacklight for the future

[Naomi Dushay]

Sorry for the cross-posting and the length; I think this is an  
important topic.

Back in November, I visited UVA to talk to talk about solrmarc and  
Blacklight.  While I was there, Bob Haschart and Matt Mitchell and I  
had a discussion about the need to separate general code from site- 
specific code for digital library OSS projects.  My personal  
experience is with the VuFind, Blacklight and solrmarc projects;  my  
words below may or may not be applicable to other projects.

Rationale:
---
If we don't separate general code from site-specific code for our OSS  
projects, then every site using the software creates an individual  
implementation silo when they customize the code.    If we all have  
separate silos, then sharing our fixes and our improvements has too  
many hurdles and doesn't happen.  For example, we at Stanford have  
customized the VuFind code.  We haven't been able to cleanly separate  
our local mods from the general code, so 1) we haven't pulled down a  
new version of the VuFind project code since August b/c we know it  
will be a job synching up   and  2) we haven't shared our fixes  
because we need to ensure we aren't inadvertently including local mods.

Thus, the silo approach will lose a great deal of potential  
contributions from the broader digital library programming community,  
which, after all, is one of the great benefits of OSS.  Yet silo  
development is hard to avoid if site-specific modifications must be  
woven into the fabric of the general code. The OSS code itself needs  
to be structured to avoid this, and needs to have an exemplar  
implementation that models how to use it.

How can separation of general from site-specific code be achieved?
---
In a broad brushstroke sense, the existing code in many of our OSS  
projects needs to be refactored for this separation.  Refactoring in  
and of itself requires a suite of tests to ensure it doesn't break  
anything.   I, for one, am not keen on three-steps-forward-two-steps- 
backward development we risk without tests.  We, as a community, need  
to practice test driven development from now on.  If it's worth  
testing once, it's worth adding it to the test suite to ensure it  
doesn't break in the future.

We believe we must use a different model for using the community  
source control (SVN) repository, based on our experience of other open  
source projects.

1.  the trunk is latest *proven working* version of the code.
2.  A "release" is a well tested (beyond "proven working") version  
warranting a "freeze"
3.  A "beta" is a candidate for release - it is "proven working" and  
we want in situ tests before we consider it a release.
3.  All unstable development occurs in a branch, not the trunk.

This approach requires excellent automated test suites and continuous  
integration.  (Hmmm ... test driven development ...)

We need to create separate source control projects for community used  
site-specific code.

Within the general project code, we need to create exemplars of a site- 
specific implementation as a pattern for other developers to copy.    
This might be achieved via techniques such as configuration files and  
overridden classes (think java classpaths to grab the local version  
first, or include paths for Ruby or PHP).  And we need to document how  
to do this.  The point has already been raised that we may want to  
exemplars for both individual sites and consortial sites.


What are the next steps, specifically?

1.  Get comments on this approach from the community.
Is there someplace beyond this list that we should be polling?

2.  Create suite of automatable tests, and ant tasks to run them.
There is *Unit test support for Ruby, as far as I know.  As for web in- 
situ testing, we are using Selenium in our current VuFind instance.

3.  We need to do a much better job of splitting out (in source  
control) exemplar site specific code (e.g. UVa's specifics for  
Blacklight) from the general code base.
We need comments on ways that you were compelled to branch off from  
the current code.  Clearly, every possibility can't be addressed in  
the general code base ... but we can probably do better than we are now.

4.  We need to ensure that non-backwards compatible changes to the  
general code won't break site specific installations:  this should be  
achievable with good beta and release tagging practices and exemplars  
on how to customize for your site without mucking about with the  
general code.

5.  Notify the community of our specific plans.

6.  Get the major "customers" squared away with a "release" version of  
the current code included in their projects, before the source control  
refactoring.

7.  Reorganize the "current" SVN code as needed.
7a. Take site specific code out of the general case
7b. Add cleanly separated exemplar implementation(s)
7c. Ensure current trunk is working (test suites for generic code and  
for exemplar implementation)
7d. Create release, beta and development branches as needed
7e. Document document document

8.  Set up continuous integration for trunk and release commits.


Naomi Dushay
ndushay at stanford.edu