From m.e.phillips at durham.ac.uk Wed Dec 5 17:44:04 2012 From: m.e.phillips at durham.ac.uk (PHILLIPS M.E.) Date: Wed, 5 Dec 2012 17:44:04 +0000 Subject: [Umlaut-general] Umlaut 3 and semicolons Message-ID: Hi, We're back at work having a got at getting Umlaut 3 up and running as a front end to Innovative Interfaces' WebBridge. We've been making good progress and my colleague Sebastian really appreciated the ease of installation now it's a Gem. We noticed some problems with some of our test examples, particularly book records from COPAC, the UK union catalogue: http://www.copac.ac.uk/ Many of them have (or had - they may have fixed this) unescaped semicolons in the URLs, for example in the "pages" field. So you get stuff like this: ?ctx_ver=Z39.88-2004&rfr_id=info:sid/mimas.ac.uk:copac&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.isbn=0077107756&rft.genre=book&rft.btitle=Economics&rft.au=Begg,+David+K.+H.&rft.au=Fischer,+Stanley.&rft.au=Dornbusch,+Rudiger.&rft.place=London+:&rft.pub=McGraw-Hill+Education,&rft.date=2005&rft.edition=8th+ed.&rft.pages=xxx,+674+p.+:+ill.+&+27+cm. Splitting that apart we have: [ 27 cm.] => [ctx_ver] => Z39.88-2004 [rfr_id] => info:sid/mimas.ac.uk:copac [rft.au] => Array ( [0] => Begg, David K. H. [1] => Fischer, Stanley. [2] => Dornbusch, Rudiger. ) [rft.btitle] => Economics [rft.date] => 2005 [rft.edition] => 8th ed. [rft.genre] => book [rft.isbn] => 0077107756 [rft.pages] => xxx, 674 p. : ill. [rft.place] => London : [rft.pub] => McGraw-Hill Education, [rft_val_fmt] => info:ofi/fmt:kev:mtx:book (this is from a PHP script for analysing OpenURLs). You will see that, splitting at the ampersands and semicolons, it has produced an extra "field" called "27 cm." with no value, i.e. there isn't even an equals sign after the name of the field. When we throw this at our Umlaut and at Johns Hopkins, we get an error. However NYU and Vanderbilt don't have problems with it. Are you on an older version of Umlaut? Here's the various URLs as shortened things: Johns Hopkins: http://bit.ly/TJNC21 NYU: http://bit.ly/TJNDTI Vanderbilt: http://bit.ly/WJzgzc (No link to ours as it's behind a firewall at present.) (Actually, I think the above examples have an ampersand, somehow, but we've mainly come across the issue with unescaped semicolons.) Can this be made more robust? Matthew -- Matthew Phillips Electronic Systems Librarian, Durham University Durham University Library, Stockton Road, Durham, DH1 3LY +44 (0)191 334 2941 From rossfsinger at gmail.com Wed Dec 5 19:08:43 2012 From: rossfsinger at gmail.com (Ross Singer) Date: Wed, 5 Dec 2012 14:08:43 -0500 Subject: [Umlaut-general] Umlaut 3 and semicolons In-Reply-To: References: Message-ID: <26B5AC44-F04F-49FE-B033-AB5973D2B375@gmail.com> Your unescaped semicolons are going to be a problem, since they're illegal. Ampersands and semicolons are synonymous in the query strings (i.e. they're both parameter delimiters), so Umlaut is likely not the only service that could be choking on this. Regarding why JHU crashes while NYU and Vandy don't could be because they're using different servers (different version of Umlaut, too, perhaps). JHU uses Passenger while I think NYU uses a thing cluster? Don't know, but I would contact COPAC and tell them they need to escape their openurl values. -Ross. On Dec 5, 2012, at 12:44 PM, "PHILLIPS M.E." wrote: > Hi, > > We're back at work having a got at getting Umlaut 3 up and running as a front end to Innovative Interfaces' WebBridge. We've been making good progress and my colleague Sebastian really appreciated the ease of installation now it's a Gem. > > We noticed some problems with some of our test examples, particularly book records from COPAC, the UK union catalogue: http://www.copac.ac.uk/ > > Many of them have (or had - they may have fixed this) unescaped semicolons in the URLs, for example in the "pages" field. So you get stuff like this: > > ?ctx_ver=Z39.88-2004&rfr_id=info:sid/mimas.ac.uk:copac&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.isbn=0077107756&rft.genre=book&rft.btitle=Economics&rft.au=Begg,+David+K.+H.&rft.au=Fischer,+Stanley.&rft.au=Dornbusch,+Rudiger.&rft.place=London+:&rft.pub=McGraw-Hill+Education,&rft.date=2005&rft.edition=8th+ed.&rft.pages=xxx,+674+p.+:+ill.+&+27+cm. > > Splitting that apart we have: > > [ 27 cm.] => > [ctx_ver] => Z39.88-2004 > [rfr_id] => info:sid/mimas.ac.uk:copac > [rft.au] => Array > ( > [0] => Begg, David K. H. > [1] => Fischer, Stanley. > [2] => Dornbusch, Rudiger. > ) > > [rft.btitle] => Economics > [rft.date] => 2005 > [rft.edition] => 8th ed. > [rft.genre] => book > [rft.isbn] => 0077107756 > [rft.pages] => xxx, 674 p. : ill. > [rft.place] => London : > [rft.pub] => McGraw-Hill Education, > [rft_val_fmt] => info:ofi/fmt:kev:mtx:book > > (this is from a PHP script for analysing OpenURLs). You will see that, splitting at the ampersands and semicolons, it has produced an extra "field" called "27 cm." with no value, i.e. there isn't even an equals sign after the name of the field. > > When we throw this at our Umlaut and at Johns Hopkins, we get an error. However NYU and Vanderbilt don't have problems with it. Are you on an older version of Umlaut? > > Here's the various URLs as shortened things: > > Johns Hopkins: http://bit.ly/TJNC21 > NYU: http://bit.ly/TJNDTI > Vanderbilt: http://bit.ly/WJzgzc > > (No link to ours as it's behind a firewall at present.) > > (Actually, I think the above examples have an ampersand, somehow, but we've mainly come across the issue with unescaped semicolons.) > > Can this be made more robust? > > Matthew > > -- > Matthew Phillips > Electronic Systems Librarian, Durham University > Durham University Library, Stockton Road, Durham, DH1 3LY > +44 (0)191 334 2941 > > > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general From rochkind at jhu.edu Wed Dec 5 19:25:02 2012 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Wed, 05 Dec 2012 14:25:02 -0500 Subject: [Umlaut-general] Umlaut 3 and semicolons In-Reply-To: <26B5AC44-F04F-49FE-B033-AB5973D2B375@gmail.com> References: <26B5AC44-F04F-49FE-B033-AB5973D2B375@gmail.com> Message-ID: <50BF9F8E.5020006@jhu.edu> Alternately, you can look into doing something at the apache layer to rewrite those URLs to be legal, without messing up other URLs, before it reaches any part of the Rails stack. It should be possible, but you're on your own there. On 12/5/2012 2:08 PM, Ross Singer wrote: > Your unescaped semicolons are going to be a problem, since they're illegal. Ampersands and semicolons are synonymous in the query strings (i.e. they're both parameter delimiters), so Umlaut is likely not the only service that could be choking on this. > > Regarding why JHU crashes while NYU and Vandy don't could be because they're using different servers (different version of Umlaut, too, perhaps). JHU uses Passenger while I think NYU uses a thing cluster? > > Don't know, but I would contact COPAC and tell them they need to escape their openurl values. > > -Ross. > > On Dec 5, 2012, at 12:44 PM, "PHILLIPS M.E." wrote: > >> Hi, >> >> We're back at work having a got at getting Umlaut 3 up and running as a front end to Innovative Interfaces' WebBridge. We've been making good progress and my colleague Sebastian really appreciated the ease of installation now it's a Gem. >> >> We noticed some problems with some of our test examples, particularly book records from COPAC, the UK union catalogue: http://www.copac.ac.uk/ >> >> Many of them have (or had - they may have fixed this) unescaped semicolons in the URLs, for example in the "pages" field. So you get stuff like this: >> >> ?ctx_ver=Z39.88-2004&rfr_id=info:sid/mimas.ac.uk:copac&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.isbn=0077107756&rft.genre=book&rft.btitle=Economics&rft.au=Begg,+David+K.+H.&rft.au=Fischer,+Stanley.&rft.au=Dornbusch,+Rudiger.&rft.place=London+:&rft.pub=McGraw-Hill+Education,&rft.date=2005&rft.edition=8th+ed.&rft.pages=xxx,+674+p.+:+ill.+&+27+cm. >> >> Splitting that apart we have: >> >> [ 27 cm.] => >> [ctx_ver] => Z39.88-2004 >> [rfr_id] => info:sid/mimas.ac.uk:copac >> [rft.au] => Array >> ( >> [0] => Begg, David K. H. >> [1] => Fischer, Stanley. >> [2] => Dornbusch, Rudiger. >> ) >> >> [rft.btitle] => Economics >> [rft.date] => 2005 >> [rft.edition] => 8th ed. >> [rft.genre] => book >> [rft.isbn] => 0077107756 >> [rft.pages] => xxx, 674 p. : ill. >> [rft.place] => London : >> [rft.pub] => McGraw-Hill Education, >> [rft_val_fmt] => info:ofi/fmt:kev:mtx:book >> >> (this is from a PHP script for analysing OpenURLs). You will see that, splitting at the ampersands and semicolons, it has produced an extra "field" called "27 cm." with no value, i.e. there isn't even an equals sign after the name of the field. >> >> When we throw this at our Umlaut and at Johns Hopkins, we get an error. However NYU and Vanderbilt don't have problems with it. Are you on an older version of Umlaut? >> >> Here's the various URLs as shortened things: >> >> Johns Hopkins: http://bit.ly/TJNC21 >> NYU: http://bit.ly/TJNDTI >> Vanderbilt: http://bit.ly/WJzgzc >> >> (No link to ours as it's behind a firewall at present.) >> >> (Actually, I think the above examples have an ampersand, somehow, but we've mainly come across the issue with unescaped semicolons.) >> >> Can this be made more robust? >> >> Matthew >> >> -- >> Matthew Phillips >> Electronic Systems Librarian, Durham University >> Durham University Library, Stockton Road, Durham, DH1 3LY >> +44 (0)191 334 2941 >> >> >> _______________________________________________ >> Umlaut-general mailing list >> Umlaut-general at rubyforge.org >> http://rubyforge.org/mailman/listinfo/umlaut-general > > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general > From rochkind at jhu.edu Wed Dec 5 18:48:20 2012 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Wed, 05 Dec 2012 13:48:20 -0500 Subject: [Umlaut-general] Umlaut 3 and semicolons In-Reply-To: References: Message-ID: <50BF96F4.3020104@jhu.edu> I am not on an older version of Umlaut, I'm running the latest 3.x release, running on Passenger. I bet NYU is running Umlaut 2.x still. I'm not sure what's going on. Would need more information about the nature of "the error" what happens how. Also curious if the URLs generated by your external source are just plain illegal URLs. The old version of Umlaut came with a perl script that could be used at the apache level to turn certain kinds of illegal URLs into legal URLs before it hit "Mongrel", which was the software the old Umlaut used as an app server. Mongrel would choke on illegal URLs, but external sources sent em anyway. This was a different sort of illegal URL, sent by EBSCO, involving unescaped brackets, not semi-colons. I took this out of Umlaut 3.x, because it seemed to be no longer neccesary with Passenger. But you are running into a slightly different problem. However, if they are indeed illegal URLs, you might need to use the same sort of technique, installing a filter at the apache level to 'correct' them before they reach passenger/umlaut. I'm not sure. Again, I'm just guessing randomly here, as you've given us no information about the nature of your error. On 12/5/2012 12:44 PM, PHILLIPS M.E. wrote: > Hi, > > We're back at work having a got at getting Umlaut 3 up and running as > a front end to Innovative Interfaces' WebBridge. We've been making > good progress and my colleague Sebastian really appreciated the ease > of installation now it's a Gem. > > We noticed some problems with some of our test examples, particularly > book records from COPAC, the UK union catalogue: > http://www.copac.ac.uk/ > > Many of them have (or had - they may have fixed this) unescaped > semicolons in the URLs, for example in the "pages" field. So you get > stuff like this: > > ?ctx_ver=Z39.88-2004&rfr_id=info:sid/mimas.ac.uk:copac&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.isbn=0077107756&rft.genre=book&rft.btitle=Economics&rft.au=Begg,+David+K.+H.&rft.au=Fischer,+Stanley.&rft.au=Dornbusch,+Rudiger.&rft.place=London+:&rft.pub=McGraw-Hill+Education,&rft.date=2005&rft.edition=8th+ed.&rft.pages=xxx,+674+p.+:+ill.+&+27+cm. > > Splitting that apart we have: > > [ 27 cm.] => [ctx_ver] => Z39.88-2004 [rfr_id] => > info:sid/mimas.ac.uk:copac [rft.au] => Array ( [0] => Begg, David K. > H. [1] => Fischer, Stanley. [2] => Dornbusch, Rudiger. ) > > [rft.btitle] => Economics [rft.date] => 2005 [rft.edition] => 8th > ed. [rft.genre] => book [rft.isbn] => 0077107756 [rft.pages] => xxx, > 674 p. : ill. [rft.place] => London : [rft.pub] => McGraw-Hill > Education, [rft_val_fmt] => info:ofi/fmt:kev:mtx:book > > (this is from a PHP script for analysing OpenURLs). You will see > that, splitting at the ampersands and semicolons, it has produced an > extra "field" called "27 cm." with no value, i.e. there isn't even an > equals sign after the name of the field. > > When we throw this at our Umlaut and at Johns Hopkins, we get an > error. However NYU and Vanderbilt don't have problems with it. Are > you on an older version of Umlaut? > > Here's the various URLs as shortened things: > > Johns Hopkins: http://bit.ly/TJNC21 NYU: http://bit.ly/TJNDTI > Vanderbilt: http://bit.ly/WJzgzc > > (No link to ours as it's behind a firewall at present.) > > (Actually, I think the above examples have an ampersand, somehow, but > we've mainly come across the issue with unescaped semicolons.) > > Can this be made more robust? > > Matthew > From std5 at nyu.edu Wed Dec 5 19:18:57 2012 From: std5 at nyu.edu (Scot Dalton) Date: Wed, 5 Dec 2012 14:18:57 -0500 Subject: [Umlaut-general] Umlaut 3 and semicolons In-Reply-To: <26B5AC44-F04F-49FE-B033-AB5973D2B375@gmail.com> References: <26B5AC44-F04F-49FE-B033-AB5973D2B375@gmail.com> Message-ID: <8750FEBD-DBA0-4083-B1FB-0A8BCE763D8C@nyu.edu> On Dec 5, 2012, at Dec 5, 2:08 PM, Ross Singer wrote: > Regarding why JHU crashes while NYU and Vandy don't could be because they're using different servers (different version of Umlaut, too, perhaps). JHU uses Passenger while I think NYU uses a thing cluster? NYU is using Passenger now but we haven't yet upgraded to Umlaut3. Thanks, Scot -- Scot Dalton Phone: (212) 998-2674 Web Services Division of Libraries New York University From dale.poulter at Vanderbilt.Edu Wed Dec 5 20:30:29 2012 From: dale.poulter at Vanderbilt.Edu (Poulter, Dale) Date: Wed, 5 Dec 2012 20:30:29 +0000 Subject: [Umlaut-general] Umlaut 3 and semicolons In-Reply-To: <8750FEBD-DBA0-4083-B1FB-0A8BCE763D8C@nyu.edu> References: <26B5AC44-F04F-49FE-B033-AB5973D2B375@gmail.com> <8750FEBD-DBA0-4083-B1FB-0A8BCE763D8C@nyu.edu> Message-ID: <4B55C1D28471794C811F425520B7B22506ACE3@ITS-HCWNEM104.ds.vanderbilt.edu> Vanderbilt is using Passenger on Umlaut v3. -----Original Message----- From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Scot Dalton Sent: Wednesday, December 05, 2012 1:19 PM To: umlaut-general at rubyforge.org Subject: Re: [Umlaut-general] Umlaut 3 and semicolons On Dec 5, 2012, at Dec 5, 2:08 PM, Ross Singer wrote: > Regarding why JHU crashes while NYU and Vandy don't could be because they're using different servers (different version of Umlaut, too, perhaps). JHU uses Passenger while I think NYU uses a thing cluster? NYU is using Passenger now but we haven't yet upgraded to Umlaut3. Thanks, Scot -- Scot Dalton Phone: (212) 998-2674 Web Services Division of Libraries New York University _______________________________________________ Umlaut-general mailing list Umlaut-general at rubyforge.org http://rubyforge.org/mailman/listinfo/umlaut-general From m.e.phillips at durham.ac.uk Thu Dec 6 12:25:50 2012 From: m.e.phillips at durham.ac.uk (PHILLIPS M.E.) Date: Thu, 6 Dec 2012 12:25:50 +0000 Subject: [Umlaut-general] Umlaut 3 and semicolons In-Reply-To: <50BF96F4.3020104@jhu.edu> References: <50BF96F4.3020104@jhu.edu> Message-ID: Yes, you're right, the URLs were in effect illegal owing to the semicolons not being escaped. I think actually they were quite old examples and COPAC has fixed the issue at their end. It's probably not worth bothering with unless I find other real-life systems doing it. The error is easy to reproduce without semicolons, however. All it takes is for there to be a query parameter that has no value, not even an equals sign. So http://palimpsest-vm1.dur.ac.uk:3000/resolve?foo=&bar= causes no trouble, but http://palimpsest-vm1.dur.ac.uk:3000/resolve?foo&bar= provokes the error. The Application Trace is empty, which I guess means it's nothing to do with Umlaut, and the framework trace starts: ensure_valid_encoding (0.5.2) lib/ensure_valid_encoding.rb:73:in `ensure_valid_encoding!' umlaut (3.0.4) app/controllers/resolve_controller.rb:161:in `block in init_processing' umlaut (3.0.4) app/controllers/resolve_controller.rb:160:in `each' umlaut (3.0.4) app/controllers/resolve_controller.rb:160:in `init_processing' activesupport (3.2.9) lib/active_support/callbacks.rb:418:in `_run__2790019675456604458__process_action__1500112367114179645__callbacks' activesupport (3.2.9) lib/active_support/callbacks.rb:405:in `__run_callback' etc. The error says undefined method `replace' for nil:NilClass The request parameters are shown as: {"foo"=>nil, "bar"=>""} It must be doing some operation on all the parameter values (to do with character encoding, it looks like) and comes a cropper when it hits the nil value for the foo parameter. If I get more real-life examples, I should be able to fix, as all our OpenURLs get redirected via a script under my control, and I could use heuristic methods to guess whether the semicolons are parameter separators or unescaped parts of parameter values. Matthew From rochkind at jhu.edu Thu Dec 6 14:16:36 2012 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 6 Dec 2012 14:16:36 +0000 Subject: [Umlaut-general] Umlaut 3 and semicolons In-Reply-To: References: <50BF96F4.3020104@jhu.edu>, Message-ID: <665DBC51D0250A47B4F9306CE71E5FB776C60E57@JHEMTEBEX1.win.ad.jhu.edu> Actually, I think that is a bug in Umlaut, thanks for the stack trace! The ensure_valid_encoding thing, Umlaut is passing a nil value too, when ensure_valid_encoding can't handle nil values, it looks like. I'll try to reproduce and fix, thanks! (I can only speak for the one you provided a stack trace for, I don't know if the semi-colon one is the same) ________________________________________ From: umlaut-general-bounces at rubyforge.org [umlaut-general-bounces at rubyforge.org] on behalf of PHILLIPS M.E. [m.e.phillips at durham.ac.uk] Sent: Thursday, December 06, 2012 7:25 AM To: umlaut-general at rubyforge.org Subject: Re: [Umlaut-general] Umlaut 3 and semicolons Yes, you're right, the URLs were in effect illegal owing to the semicolons not being escaped. I think actually they were quite old examples and COPAC has fixed the issue at their end. It's probably not worth bothering with unless I find other real-life systems doing it. The error is easy to reproduce without semicolons, however. All it takes is for there to be a query parameter that has no value, not even an equals sign. So http://palimpsest-vm1.dur.ac.uk:3000/resolve?foo=&bar= causes no trouble, but http://palimpsest-vm1.dur.ac.uk:3000/resolve?foo&bar= provokes the error. The Application Trace is empty, which I guess means it's nothing to do with Umlaut, and the framework trace starts: ensure_valid_encoding (0.5.2) lib/ensure_valid_encoding.rb:73:in `ensure_valid_encoding!' umlaut (3.0.4) app/controllers/resolve_controller.rb:161:in `block in init_processing' umlaut (3.0.4) app/controllers/resolve_controller.rb:160:in `each' umlaut (3.0.4) app/controllers/resolve_controller.rb:160:in `init_processing' activesupport (3.2.9) lib/active_support/callbacks.rb:418:in `_run__2790019675456604458__process_action__1500112367114179645__callbacks' activesupport (3.2.9) lib/active_support/callbacks.rb:405:in `__run_callback' etc. The error says undefined method `replace' for nil:NilClass The request parameters are shown as: {"foo"=>nil, "bar"=>""} It must be doing some operation on all the parameter values (to do with character encoding, it looks like) and comes a cropper when it hits the nil value for the foo parameter. If I get more real-life examples, I should be able to fix, as all our OpenURLs get redirected via a script under my control, and I could use heuristic methods to guess whether the semicolons are parameter separators or unescaped parts of parameter values. Matthew _______________________________________________ Umlaut-general mailing list Umlaut-general at rubyforge.org http://rubyforge.org/mailman/listinfo/umlaut-general From rochkind at jhu.edu Thu Dec 6 21:58:20 2012 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 06 Dec 2012 16:58:20 -0500 Subject: [Umlaut-general] Umlaut 3 and semicolons In-Reply-To: <665DBC51D0250A47B4F9306CE71E5FB776C60E57@JHEMTEBEX1.win.ad.jhu.edu> References: <50BF96F4.3020104@jhu.edu> <665DBC51D0250A47B4F9306CE71E5FB776C60E57@JHEMTEBEX1.win.ad.jhu.edu> Message-ID: <50C114FC.8010700@jhu.edu> Okay, I fixed this by fixing ensure_valid_encoding (a gem I wrote). Thanks for the bug report, Matthew. No changes to Umlaut; to get the fix you need to update to latest version of ensure_valid_encoding, by just running: bundle update ensure_valid_encoding and then check your updated Gemfile.lock into your source control (if you're using source control, which you should be). ensure_valid_encoding now allows nil as an input parameter, just returning nil, because this makes sense and makes it's use more convenient. To be clear: With this fix, umlaut won't choke on URLs of the form "&foo&bar=2" anymore -- with a key, but no equal sign and value. I can't promise it will do the 'right thing' with weird URLs that don't actually mean what they're intended to mean, but it won't raise. Didn't try a variation of your semicolon-holding URLs, if you have one that still raises please supply it, along with stack trace, and I can see if I can fix that too (feel free to submit as an Issue in the github project). On 12/6/2012 9:16 AM, Jonathan Rochkind wrote: > Actually, I think that is a bug in Umlaut, thanks for the stack > trace! > > The ensure_valid_encoding thing, Umlaut is passing a nil value too, > when ensure_valid_encoding can't handle nil values, it looks like. > > I'll try to reproduce and fix, thanks! > > (I can only speak for the one you provided a stack trace for, I don't > know if the semi-colon one is the same) > ________________________________________ From: > umlaut-general-bounces at rubyforge.org > [umlaut-general-bounces at rubyforge.org] on behalf of PHILLIPS M.E. > [m.e.phillips at durham.ac.uk] Sent: Thursday, December 06, 2012 7:25 > AM To: umlaut-general at rubyforge.org Subject: Re: [Umlaut-general] > Umlaut 3 and semicolons > > Yes, you're right, the URLs were in effect illegal owing to the > semicolons not being escaped. I think actually they were quite old > examples and COPAC has fixed the issue at their end. It's probably > not worth bothering with unless I find other real-life systems doing > it. > > The error is easy to reproduce without semicolons, however. All it > takes is for there to be a query parameter that has no value, not > even an equals sign. So > > http://palimpsest-vm1.dur.ac.uk:3000/resolve?foo=&bar= > > causes no trouble, but > > http://palimpsest-vm1.dur.ac.uk:3000/resolve?foo&bar= > > provokes the error. The Application Trace is empty, which I guess > means it's nothing to do with Umlaut, and the framework trace > starts: > > ensure_valid_encoding (0.5.2) lib/ensure_valid_encoding.rb:73:in > `ensure_valid_encoding!' umlaut (3.0.4) > app/controllers/resolve_controller.rb:161:in `block in > init_processing' umlaut (3.0.4) > app/controllers/resolve_controller.rb:160:in `each' umlaut (3.0.4) > app/controllers/resolve_controller.rb:160:in `init_processing' > activesupport (3.2.9) lib/active_support/callbacks.rb:418:in > `_run__2790019675456604458__process_action__1500112367114179645__callbacks' > > activesupport (3.2.9) lib/active_support/callbacks.rb:405:in `__run_callback' > etc. > > The error says > > undefined method `replace' for nil:NilClass > > The request parameters are shown as: > > {"foo"=>nil, "bar"=>""} > > It must be doing some operation on all the parameter values (to do > with character encoding, it looks like) and comes a cropper when it > hits the nil value for the foo parameter. > > If I get more real-life examples, I should be able to fix, as all our > OpenURLs get redirected via a script under my control, and I could > use heuristic methods to guess whether the semicolons are parameter > separators or unescaped parts of parameter values. > > Matthew > > _______________________________________________ Umlaut-general > mailing list Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general > _______________________________________________ Umlaut-general > mailing list Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general > From m.e.phillips at durham.ac.uk Mon Dec 10 16:52:44 2012 From: m.e.phillips at durham.ac.uk (PHILLIPS M.E.) Date: Mon, 10 Dec 2012 16:52:44 +0000 Subject: [Umlaut-general] E-mail export Message-ID: Hello, One of the services in Umlaut we tried enabling with EmailExport. The option appears in the right-hand panel, and opens a box showing the title of the article, and allowing me to enter my e-mail address. Strangely on clicking Send the box is redisplayed, but with the article title now appearing as the box title, with a further box for the e-mail address, and another Send button. On clicking that, the box redisplays again, with no information apart from the box for the e-mail address. Nothing really happens after that, and we do not get an e-mail. See http://bit.ly/RX4Ugl for an example on our test server (which should be accessible to the world now). This also happens on Vanderbilt's server (not sure if this is your production service): http://umlaut.library.vanderbilt.edu/go/1409083 On NYU it works fine: https://getit.library.nyu.edu/go/6280370 and I got an e-mail from you. Johns Hopkins does not offer the option. I suspect I am missing some configuration somewhere, as we have not specified an origin e-mail address anywhere. The email_export.rb file is rather brief: [dul0zz56 at palimpsest-vm1 ~]$ more ./.rvm/gems/ruby-1.9.3-p327 at umlaut/gems/umlaut-3.0.4/lib/service_adaptors/email_export.rb class EmailExport < AjaxExport def initialize(config) @display_text ||= "Email" @form_controller ||= "export_email" @form_action ||= "email" super(config) end end It does not give me many clues. I have found some stuff under /gems/umlaut-3.0.4/app/views/export_email which looks like where it's all coming from. Can't see anything obvious in documentation. Sorry if I've overlooked something. I realise folk may be busy, so please note this is not urgent in any way: just me making a note in public of something I don't understand yet. I have plenty of other things to explore in the meantime so unless anyone has a quick pointer to where I can read more, I'll leave this for a bit and come back to it. -- Matthew Phillips Electronic Systems Librarian, Durham University Durham University Library, Stockton Road, Durham, DH1 3LY +44 (0)191 334 2941 From m.e.phillips at durham.ac.uk Mon Dec 10 17:13:25 2012 From: m.e.phillips at durham.ac.uk (PHILLIPS M.E.) Date: Mon, 10 Dec 2012 17:13:25 +0000 Subject: [Umlaut-general] E-mail export In-Reply-To: <50C61510.3060305@jhu.edu> References: <50C61510.3060305@jhu.edu> Message-ID: We have only used one environment at present. We are just running on WEBrick, not Passenger or anything that could be productionalised. We have not consciously done any configuration of e-mail details, like From address, SMTP server, etc. Matthew -----Original Message----- From: Jonathan Rochkind [mailto:rochkind at jhu.edu] Sent: 10 December 2012 17:00 To: umlaut-general at rubyforge.org Cc: PHILLIPS M.E. Subject: Re: [Umlaut-general] E-mail export Scot, do you use that feature? I confess I've NEVER used it --- I have found that OpenURLs sent to the resolver just don't have enough citation details to have useful citation export. So it's possible it broke in Umlaut 3, if nobody else uses it. We can get it working again, one way or another. I might not have time to look at it for a bit myself though. Although it also sounds like it's possible you've run into a gotcha with Rails in general and JS -- have you been switching between production and development mode, in the same on-disk rails instance, and at the same URL? I have found that this sometimes confuses Rails3 as to your assets, and causes your browser to load two copies of the JS. The bheavior you mention sounds like it could be a "two copies of JS" issue. Delete your ./public/assets, delete your ./tmp/cache, and delete your browser file cache in the browser. (Then recompile assets if you are going to be running in production). See if that changes things. And try not to switch between dev and production mode in the same instance and url. On 12/10/2012 11:52 AM, PHILLIPS M.E. wrote: > Hello, > > One of the services in Umlaut we tried enabling with EmailExport. > The option appears in the right-hand panel, and opens a box showing > the title of the article, and allowing me to enter my e-mail address. > Strangely on clicking Send the box is redisplayed, but with the > article title now appearing as the box title, with a further box for > the e-mail address, and another Send button. On clicking that, the box > redisplays again, with no information apart from the box for the > e-mail address. Nothing really happens after that, and we do not get > an e-mail. > > See http://bit.ly/RX4Ugl for an example on our test server (which > should be accessible to the world now). > > This also happens on Vanderbilt's server (not sure if this is your > production service): http://umlaut.library.vanderbilt.edu/go/1409083 > > On NYU it works fine: https://getit.library.nyu.edu/go/6280370 and I > got an e-mail from you. > > Johns Hopkins does not offer the option. > > I suspect I am missing some configuration somewhere, as we have not > specified an origin e-mail address anywhere. The email_export.rb file > is rather brief: > > [dul0zz56 at palimpsest-vm1 ~]$ more > ./.rvm/gems/ruby-1.9.3-p327 at umlaut/gems/umlaut-3.0.4/lib/service_adapt > ors/email_export.rb > > class EmailExport < AjaxExport > > def initialize(config) @display_text ||= "Email" @form_controller ||= > "export_email" @form_action ||= "email" super(config) end > > end > > It does not give me many clues. I have found some stuff under > /gems/umlaut-3.0.4/app/views/export_email which looks like where it's > all coming from. Can't see anything obvious in documentation. Sorry > if I've overlooked something. > > I realise folk may be busy, so please note this is not urgent in any > way: just me making a note in public of something I don't understand > yet. I have plenty of other things to explore in the meantime so > unless anyone has a quick pointer to where I can read more, I'll leave > this for a bit and come back to it. > > From std5 at nyu.edu Mon Dec 10 17:35:41 2012 From: std5 at nyu.edu (Scot Dalton) Date: Mon, 10 Dec 2012 12:35:41 -0500 Subject: [Umlaut-general] E-mail export In-Reply-To: References: <50C61510.3060305@jhu.edu> Message-ID: We use it, but only its only in production for version 2. I seem to remember encountering a bug in it in v3, and thought I pushed the fix to master. I'll have a look, but probably not until tomorrow. Thanks, Scot On Dec 10, 2012, at Dec 10, 12:13 PM, PHILLIPS M.E. wrote: > We have only used one environment at present. We are just running on WEBrick, not Passenger or anything that could be productionalised. We have not consciously done any configuration of e-mail details, like From address, SMTP server, etc. > > Matthew > > -----Original Message----- > From: Jonathan Rochkind [mailto:rochkind at jhu.edu] > Sent: 10 December 2012 17:00 > To: umlaut-general at rubyforge.org > Cc: PHILLIPS M.E. > Subject: Re: [Umlaut-general] E-mail export > > Scot, do you use that feature? > > I confess I've NEVER used it --- I have found that OpenURLs sent to the resolver just don't have enough citation details to have useful citation export. > > So it's possible it broke in Umlaut 3, if nobody else uses it. We can get it working again, one way or another. I might not have time to look at it for a bit myself though. > > Although it also sounds like it's possible you've run into a gotcha with Rails in general and JS -- have you been switching between production and development mode, in the same on-disk rails instance, and at the same URL? I have found that this sometimes confuses Rails3 as to your assets, and causes your browser to load two copies of the JS. The bheavior you mention sounds like it could be a "two copies of JS" issue. > Delete your ./public/assets, delete your ./tmp/cache, and delete your browser file cache in the browser. (Then recompile assets if you are going to be running in production). See if that changes things. And try not to switch between dev and production mode in the same instance and url. > > On 12/10/2012 11:52 AM, PHILLIPS M.E. wrote: >> Hello, >> >> One of the services in Umlaut we tried enabling with EmailExport. >> The option appears in the right-hand panel, and opens a box showing >> the title of the article, and allowing me to enter my e-mail address. >> Strangely on clicking Send the box is redisplayed, but with the >> article title now appearing as the box title, with a further box for >> the e-mail address, and another Send button. On clicking that, the box >> redisplays again, with no information apart from the box for the >> e-mail address. Nothing really happens after that, and we do not get >> an e-mail. >> >> See http://bit.ly/RX4Ugl for an example on our test server (which >> should be accessible to the world now). >> >> This also happens on Vanderbilt's server (not sure if this is your >> production service): http://umlaut.library.vanderbilt.edu/go/1409083 >> >> On NYU it works fine: https://getit.library.nyu.edu/go/6280370 and I >> got an e-mail from you. >> >> Johns Hopkins does not offer the option. >> >> I suspect I am missing some configuration somewhere, as we have not >> specified an origin e-mail address anywhere. The email_export.rb file >> is rather brief: >> >> [dul0zz56 at palimpsest-vm1 ~]$ more >> ./.rvm/gems/ruby-1.9.3-p327 at umlaut/gems/umlaut-3.0.4/lib/service_adapt >> ors/email_export.rb >> >> > class EmailExport < AjaxExport >> >> def initialize(config) @display_text ||= "Email" @form_controller ||= >> "export_email" @form_action ||= "email" super(config) end >> >> end >> >> It does not give me many clues. I have found some stuff under >> /gems/umlaut-3.0.4/app/views/export_email which looks like where it's >> all coming from. Can't see anything obvious in documentation. Sorry >> if I've overlooked something. >> >> I realise folk may be busy, so please note this is not urgent in any >> way: just me making a note in public of something I don't understand >> yet. I have plenty of other things to explore in the meantime so >> unless anyone has a quick pointer to where I can read more, I'll leave >> this for a bit and come back to it. >> >> > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general -- Scot Dalton Phone: (212) 998-2674 Web Services Division of Libraries New York University From rochkind at jhu.edu Mon Dec 10 17:00:00 2012 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Mon, 10 Dec 2012 12:00:00 -0500 Subject: [Umlaut-general] E-mail export In-Reply-To: References: Message-ID: <50C61510.3060305@jhu.edu> Scot, do you use that feature? I confess I've NEVER used it --- I have found that OpenURLs sent to the resolver just don't have enough citation details to have useful citation export. So it's possible it broke in Umlaut 3, if nobody else uses it. We can get it working again, one way or another. I might not have time to look at it for a bit myself though. Although it also sounds like it's possible you've run into a gotcha with Rails in general and JS -- have you been switching between production and development mode, in the same on-disk rails instance, and at the same URL? I have found that this sometimes confuses Rails3 as to your assets, and causes your browser to load two copies of the JS. The bheavior you mention sounds like it could be a "two copies of JS" issue. Delete your ./public/assets, delete your ./tmp/cache, and delete your browser file cache in the browser. (Then recompile assets if you are going to be running in production). See if that changes things. And try not to switch between dev and production mode in the same instance and url. On 12/10/2012 11:52 AM, PHILLIPS M.E. wrote: > Hello, > > One of the services in Umlaut we tried enabling with EmailExport. > The option appears in the right-hand panel, and opens a box showing > the title of the article, and allowing me to enter my e-mail address. > Strangely on clicking Send the box is redisplayed, but with the > article title now appearing as the box title, with a further box for > the e-mail address, and another Send button. On clicking that, the > box redisplays again, with no information apart from the box for the > e-mail address. Nothing really happens after that, and we do not get > an e-mail. > > See http://bit.ly/RX4Ugl for an example on our test server (which > should be accessible to the world now). > > This also happens on Vanderbilt's server (not sure if this is your > production service): http://umlaut.library.vanderbilt.edu/go/1409083 > > On NYU it works fine: https://getit.library.nyu.edu/go/6280370 and I > got an e-mail from you. > > Johns Hopkins does not offer the option. > > I suspect I am missing some configuration somewhere, as we have not > specified an origin e-mail address anywhere. The email_export.rb > file is rather brief: > > [dul0zz56 at palimpsest-vm1 ~]$ more > ./.rvm/gems/ruby-1.9.3-p327 at umlaut/gems/umlaut-3.0.4/lib/service_adaptors/email_export.rb > > class EmailExport < AjaxExport > > def initialize(config) @display_text ||= "Email" @form_controller ||= > "export_email" @form_action ||= "email" super(config) end > > end > > It does not give me many clues. I have found some stuff under > /gems/umlaut-3.0.4/app/views/export_email which looks like where it's > all coming from. Can't see anything obvious in documentation. Sorry > if I've overlooked something. > > I realise folk may be busy, so please note this is not urgent in any > way: just me making a note in public of something I don't understand > yet. I have plenty of other things to explore in the meantime so > unless anyone has a quick pointer to where I can read more, I'll > leave this for a bit and come back to it. > > From std5 at nyu.edu Fri Dec 14 16:26:17 2012 From: std5 at nyu.edu (Scot Thomas Dalton) Date: Fri, 14 Dec 2012 11:26:17 -0500 Subject: [Umlaut-general] Bootstrap Branch Message-ID: Hi all, I've been working on an Umlaut interface that leverages Twitter's Bootstrap framework.[1][2] I've put a sample app on heroku.[3] I also put an app using the current style (from the master branch) up for comparison purposes.[4] I'm still working out some kinks but would love to get feedback from the community. Please email suggestions, or better still, log issues in the GitHub issues tracker.[5] Thanks, Scot [1] https://github.com/team-umlaut/umlaut/tree/bootstrap/ [2] http://twitter.github.com/bootstrap/ [3] http://umlaut-bootstrap.herokuapp.com/ [4] http://umlaut-master.herokuapp.com/ [5] https://github.com/team-umlaut/umlaut/issues -- Scot Dalton Phone: 212.998.2674 Web Development Division of Libraries New York University -------------- next part -------------- An HTML attachment was scrubbed... URL: From dale.poulter at Vanderbilt.Edu Mon Dec 17 23:49:49 2012 From: dale.poulter at Vanderbilt.Edu (Poulter, Dale) Date: Mon, 17 Dec 2012 23:49:49 +0000 Subject: [Umlaut-general] E-mail export In-Reply-To: <50C61510.3060305@jhu.edu> References: <50C61510.3060305@jhu.edu> Message-ID: <4B55C1D28471794C811F425520B7B22532FA2D@ITS-HCWNEM104.ds.vanderbilt.edu> This seems to be an issue when emailing an article but works well when emailing information about a journal (http://umlaut.library.vanderbilt.edu ). I wonder if we have a character limit? -----Original Message----- From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Jonathan Rochkind Sent: Monday, December 10, 2012 11:00 AM To: umlaut-general at rubyforge.org Subject: Re: [Umlaut-general] E-mail export Scot, do you use that feature? I confess I've NEVER used it --- I have found that OpenURLs sent to the resolver just don't have enough citation details to have useful citation export. So it's possible it broke in Umlaut 3, if nobody else uses it. We can get it working again, one way or another. I might not have time to look at it for a bit myself though. Although it also sounds like it's possible you've run into a gotcha with Rails in general and JS -- have you been switching between production and development mode, in the same on-disk rails instance, and at the same URL? I have found that this sometimes confuses Rails3 as to your assets, and causes your browser to load two copies of the JS. The bheavior you mention sounds like it could be a "two copies of JS" issue. Delete your ./public/assets, delete your ./tmp/cache, and delete your browser file cache in the browser. (Then recompile assets if you are going to be running in production). See if that changes things. And try not to switch between dev and production mode in the same instance and url. On 12/10/2012 11:52 AM, PHILLIPS M.E. wrote: > Hello, > > One of the services in Umlaut we tried enabling with EmailExport. > The option appears in the right-hand panel, and opens a box showing > the title of the article, and allowing me to enter my e-mail address. > Strangely on clicking Send the box is redisplayed, but with the > article title now appearing as the box title, with a further box for > the e-mail address, and another Send button. On clicking that, the box > redisplays again, with no information apart from the box for the > e-mail address. Nothing really happens after that, and we do not get > an e-mail. > > See http://bit.ly/RX4Ugl for an example on our test server (which > should be accessible to the world now). > > This also happens on Vanderbilt's server (not sure if this is your > production service): http://umlaut.library.vanderbilt.edu/go/1409083 > > On NYU it works fine: https://getit.library.nyu.edu/go/6280370 and I > got an e-mail from you. > > Johns Hopkins does not offer the option. > > I suspect I am missing some configuration somewhere, as we have not > specified an origin e-mail address anywhere. The email_export.rb file > is rather brief: > > [dul0zz56 at palimpsest-vm1 ~]$ more > ./.rvm/gems/ruby-1.9.3-p327 at umlaut/gems/umlaut-3.0.4/lib/service_adapt > ors/email_export.rb > > class EmailExport < AjaxExport > > def initialize(config) @display_text ||= "Email" @form_controller ||= > "export_email" @form_action ||= "email" super(config) end > > end > > It does not give me many clues. I have found some stuff under > /gems/umlaut-3.0.4/app/views/export_email which looks like where it's > all coming from. Can't see anything obvious in documentation. Sorry > if I've overlooked something. > > I realise folk may be busy, so please note this is not urgent in any > way: just me making a note in public of something I don't understand > yet. I have plenty of other things to explore in the meantime so > unless anyone has a quick pointer to where I can read more, I'll leave > this for a bit and come back to it. > > _______________________________________________ Umlaut-general mailing list Umlaut-general at rubyforge.org http://rubyforge.org/mailman/listinfo/umlaut-general From rochkind at jhu.edu Mon Dec 17 23:57:30 2012 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Mon, 17 Dec 2012 18:57:30 -0500 Subject: [Umlaut-general] E-mail export In-Reply-To: <4B55C1D28471794C811F425520B7B22532FA2D@ITS-HCWNEM104.ds.vanderbilt.edu> References: <50C61510.3060305@jhu.edu> <4B55C1D28471794C811F425520B7B22532FA2D@ITS-HCWNEM104.ds.vanderbilt.edu> Message-ID: <50CFB16A.3080402@jhu.edu> That would be odd, I can't think of any reason a character limit would be involved. But sorry, I don't have any time to debug this right now. Dale, is that something you use/need/want in your app too? Matthew, you can dive in and debug it (which would be awesome), you can refrain from using it, or you can of course right your own email export feature however you like, and easily wire it in the services.yml. Either way, anyone, please feel free to file this as an issue on Github Issues, with as much as you've been able to determine about the nature of the problem (and feel free to update more later if you figure out more!). Normally I'd say it would also be useful to make a branch on git that has a failing automated test added to it, representing the problem -- but I know making an automated test for this particular feature is going to be a nightmare (but if you come up with one, that'd be neat. :) ). On 12/17/2012 6:49 PM, Poulter, Dale wrote: > This seems to be an issue when emailing an article but works well when emailing information about a journal (http://umlaut.library.vanderbilt.edu ). I wonder if we have a character limit? > > -----Original Message----- > From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Jonathan Rochkind > Sent: Monday, December 10, 2012 11:00 AM > To: umlaut-general at rubyforge.org > Subject: Re: [Umlaut-general] E-mail export > > Scot, do you use that feature? > > I confess I've NEVER used it --- I have found that OpenURLs sent to the resolver just don't have enough citation details to have useful citation export. > > So it's possible it broke in Umlaut 3, if nobody else uses it. We can get it working again, one way or another. I might not have time to look at it for a bit myself though. > > Although it also sounds like it's possible you've run into a gotcha with Rails in general and JS -- have you been switching between production and development mode, in the same on-disk rails instance, and at the same URL? I have found that this sometimes confuses Rails3 as to your assets, and causes your browser to load two copies of the JS. The bheavior you mention sounds like it could be a "two copies of JS" issue. > Delete your ./public/assets, delete your ./tmp/cache, and delete your browser file cache in the browser. (Then recompile assets if you are going to be running in production). See if that changes things. And try not to switch between dev and production mode in the same instance and url. > > On 12/10/2012 11:52 AM, PHILLIPS M.E. wrote: >> Hello, >> >> One of the services in Umlaut we tried enabling with EmailExport. >> The option appears in the right-hand panel, and opens a box showing >> the title of the article, and allowing me to enter my e-mail address. >> Strangely on clicking Send the box is redisplayed, but with the >> article title now appearing as the box title, with a further box for >> the e-mail address, and another Send button. On clicking that, the box >> redisplays again, with no information apart from the box for the >> e-mail address. Nothing really happens after that, and we do not get >> an e-mail. >> >> See http://bit.ly/RX4Ugl for an example on our test server (which >> should be accessible to the world now). >> >> This also happens on Vanderbilt's server (not sure if this is your >> production service): http://umlaut.library.vanderbilt.edu/go/1409083 >> >> On NYU it works fine: https://getit.library.nyu.edu/go/6280370 and I >> got an e-mail from you. >> >> Johns Hopkins does not offer the option. >> >> I suspect I am missing some configuration somewhere, as we have not >> specified an origin e-mail address anywhere. The email_export.rb file >> is rather brief: >> >> [dul0zz56 at palimpsest-vm1 ~]$ more >> ./.rvm/gems/ruby-1.9.3-p327 at umlaut/gems/umlaut-3.0.4/lib/service_adapt >> ors/email_export.rb >> >> > class EmailExport < AjaxExport >> >> def initialize(config) @display_text ||= "Email" @form_controller ||= >> "export_email" @form_action ||= "email" super(config) end >> >> end >> >> It does not give me many clues. I have found some stuff under >> /gems/umlaut-3.0.4/app/views/export_email which looks like where it's >> all coming from. Can't see anything obvious in documentation. Sorry >> if I've overlooked something. >> >> I realise folk may be busy, so please note this is not urgent in any >> way: just me making a note in public of something I don't understand >> yet. I have plenty of other things to explore in the meantime so >> unless anyone has a quick pointer to where I can read more, I'll leave >> this for a bit and come back to it. >> >> > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general > > > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general >