From n.molesbenfell at gmail.com Wed Sep 5 22:55:31 2007 From: n.molesbenfell at gmail.com (Nicolai Moles-Benfell) Date: Thu, 6 Sep 2007 14:55:31 +1200 Subject: [Ruby-zoom-devel] destroy connection at end of block In-Reply-To: <763570460708280614l6ea37d1aw9c1e7d72121650d5@mail.gmail.com> References: <763570460708270725i3ab904fcy6c368b8e7398be80@mail.gmail.com> <460C1697-03E5-4A64-A91B-5AD2A4382C45@gmail.com> <763570460708280614l6ea37d1aw9c1e7d72121650d5@mail.gmail.com> Message-ID: <3E7F8F50-7328-4381-87EC-EF55C6ED4990@gmail.com> Hey, I have spent a bit of time going through the behavior of the Ruby garbage collector in regards to how it deals with RubyZoom. The good news is that by putting print statements through out RubyZoom and YAZ I can confirm that the Ruby garbage collector does close the connections with out needing to close the script, it just takes its sweet time. By 'closing the connections' I mean that the YAZ function zoom_connection_destroy is called. I tried adding a dispose method to the library that is called at the end of a block, this successfully closes the connection when called, though I have run into the problem that when the Ruby GC calls the zoom destroy method, I can not determine if the connection has already been destroyed/freed by the dispose method as I you get at this point is the pointer to the Zoom_Connection object... As far as I can tell the mark parameter of Data_Wrap_Struct is not of any help here as its purpose is to mark related objects as still being in use. Regards, Nicolai. On 29/08/2007, at 1:14 AM, Jason Ronallo wrote: > Hi, Nicolai, > > Thank you for your work on this. It's really great to have the > extended services support now. > >> The YAZ library eposes the methods ZOOM_connection_destroy, >> ZOOM_package_destroy which are being called by the Ruby garbage >> collector when destroying the Ruby connection or package objects... >> these methods can be exposed in Ruby or called explicitly at the end >> of a method that accepts a block... > > I think this is an issue of a disconnect with garbage collection. Even > explicitly calling garbage collection after the end of the block in > Ruby does not destroy the connection. > > I wonder if it has to do with the NULL in the code below. From my > reading of Ruby C-bindings where that NULL is is the position used for > marking an object so in the next sweep it is collected. Without the > object being marked I don't think ZOOM_connection_destroy is ever > doing any work. > > How to do that I don't know. As soon as it mentioned allocating memory > I backed away. :) Nicolai, would it be possible to write a mark > method? That should insure that the connection is in fact closed at > the end of a block, right? > > --Jason > > >> something like >> >> if (rb_block_given_p ()) { >> rb_yield(rb_connect); >> ZOOM_connect_destroy(cZoomConnect, NULL, >> ZOOM_connection_destroy, >> connection) >> return Qnil; >> } >> >> return rb_connection; > _______________________________________________ > Ruby-zoom-devel mailing list > Ruby-zoom-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/ruby-zoom-devel From jronallo at gmail.com Thu Sep 6 08:40:54 2007 From: jronallo at gmail.com (Jason Ronallo) Date: Thu, 6 Sep 2007 08:40:54 -0400 Subject: [Ruby-zoom-devel] destroy connection at end of block In-Reply-To: <3E7F8F50-7328-4381-87EC-EF55C6ED4990@gmail.com> References: <763570460708270725i3ab904fcy6c368b8e7398be80@mail.gmail.com> <460C1697-03E5-4A64-A91B-5AD2A4382C45@gmail.com> <763570460708280614l6ea37d1aw9c1e7d72121650d5@mail.gmail.com> <3E7F8F50-7328-4381-87EC-EF55C6ED4990@gmail.com> Message-ID: <763570460709060540s153178a9s37fed461fe94291e@mail.gmail.com> Nicolai, Thank you for looking into this further. I had no idea what was at fault here. It's good to know for certain that it's the Ruby garbage collector being slow about things. I had tried explicit garbage collection right after the end of a zoom block with no effect. Now I've moved garbage collection (GC.start) later on in my script (but not much) and the connection is closing properly. My apologies for having you do this work because I didn't understand the Ruby end of garbage collection enough. Thanks again for clearing this up for me. --Jason On 9/5/07, Nicolai Moles-Benfell wrote: > Hey, > > I have spent a bit of time going through the behavior of the Ruby > garbage collector in regards to how it deals with RubyZoom. The good > news is that by putting print statements through out RubyZoom and YAZ > I can confirm that the Ruby garbage collector does close the > connections with out needing to close the script, it just takes its > sweet time. By 'closing the connections' I mean that the YAZ function > zoom_connection_destroy is called. > > I tried adding a dispose method to the library that is called at the > end of a block, this successfully closes the connection when called, > though I have run into the problem that when the Ruby GC calls the > zoom destroy method, I can not determine if the connection has > already been destroyed/freed by the dispose method as I you get at > this point is the pointer to the Zoom_Connection object... > > As far as I can tell the mark parameter of Data_Wrap_Struct is not of > any help here as its purpose is to mark related objects as still > being in use. > > Regards, > Nicolai. > > > On 29/08/2007, at 1:14 AM, Jason Ronallo wrote: > > > Hi, Nicolai, > > > > Thank you for your work on this. It's really great to have the > > extended services support now. > > > >> The YAZ library eposes the methods ZOOM_connection_destroy, > >> ZOOM_package_destroy which are being called by the Ruby garbage > >> collector when destroying the Ruby connection or package objects... > >> these methods can be exposed in Ruby or called explicitly at the end > >> of a method that accepts a block... > > > > I think this is an issue of a disconnect with garbage collection. Even > > explicitly calling garbage collection after the end of the block in > > Ruby does not destroy the connection. > > > > I wonder if it has to do with the NULL in the code below. From my > > reading of Ruby C-bindings where that NULL is is the position used for > > marking an object so in the next sweep it is collected. Without the > > object being marked I don't think ZOOM_connection_destroy is ever > > doing any work. > > > > How to do that I don't know. As soon as it mentioned allocating memory > > I backed away. :) Nicolai, would it be possible to write a mark > > method? That should insure that the connection is in fact closed at > > the end of a block, right? > > > > --Jason > > > > > >> something like > >> > >> if (rb_block_given_p ()) { > >> rb_yield(rb_connect); > >> ZOOM_connect_destroy(cZoomConnect, NULL, > >> ZOOM_connection_destroy, > >> connection) > >> return Qnil; > >> } > >> > >> return rb_connection; > > _______________________________________________ > > Ruby-zoom-devel mailing list > > Ruby-zoom-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/ruby-zoom-devel > > _______________________________________________ > Ruby-zoom-devel mailing list > Ruby-zoom-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/ruby-zoom-devel > From walter at katipo.co.nz Mon Sep 10 21:28:15 2007 From: walter at katipo.co.nz (Walter McGinnis) Date: Tue, 11 Sep 2007 13:28:15 +1200 Subject: [Ruby-zoom-devel] Where are we in relation to a release In-Reply-To: <763570460709060540s153178a9s37fed461fe94291e@mail.gmail.com> References: <763570460708270725i3ab904fcy6c368b8e7398be80@mail.gmail.com> <460C1697-03E5-4A64-A91B-5AD2A4382C45@gmail.com> <763570460708280614l6ea37d1aw9c1e7d72121650d5@mail.gmail.com> <3E7F8F50-7328-4381-87EC-EF55C6ED4990@gmail.com> <763570460709060540s153178a9s37fed461fe94291e@mail.gmail.com> Message-ID: <1CFDA9D9-81C6-4F6B-86DA-8BB19AED9F40@katipo.co.nz> What sort of process do we now want to fulfill before releasing the next version of the gem? Have we covered all the major questions, etc? Cheers, Walter From walter at katipo.co.nz Tue Sep 11 12:04:23 2007 From: walter at katipo.co.nz (Walter McGinnis) Date: Wed, 12 Sep 2007 04:04:23 +1200 Subject: [Ruby-zoom-devel] Where are we in relation to a release In-Reply-To: <763570460709110641m27aab1c5x2321716c67a374@mail.gmail.com> References: <763570460708270725i3ab904fcy6c368b8e7398be80@mail.gmail.com> <460C1697-03E5-4A64-A91B-5AD2A4382C45@gmail.com> <763570460708280614l6ea37d1aw9c1e7d72121650d5@mail.gmail.com> <3E7F8F50-7328-4381-87EC-EF55C6ED4990@gmail.com> <763570460709060540s153178a9s37fed461fe94291e@mail.gmail.com> <1CFDA9D9-81C6-4F6B-86DA-8BB19AED9F40@katipo.co.nz> <763570460709110641m27aab1c5x2321716c67a374@mail.gmail.com> Message-ID: <7F7B1646-1E7B-47BD-A2DD-057BEA33167F@katipo.co.nz> Cool, I just wanted a round up of where we are at. The last piece for 0.4.1 sounds like documentation for extended services. You certainly have a better handle on it than I do. Let me know if I can do anything to help. On a tangental note, Kete (http://kete.net.nz/), has won the 3M award for innovation in libraries at the LIANZA conference in New Zealand. Kete uses ruby-zoom via acts_as_zoom to populate and query Zebra databases as you may remember. Cheers, Walter On Sep 12, 2007, at 1:41 AM, Jason Ronallo wrote: > First, the 0.4.0 version is currently available from rubyforge though > it doesn't show up on the rubyforge webpage. Since at least one bug > fix has been submitted since that release, I'd like to see the next > release as 0.4.1. > > I'd like to see documentation and update of the website, at least the > addition of extended services. I tried writing an rdoc rake task but > kept getting errors about not knowing the enclosing module. This seems > to be a problem with rdoc not knowing how to deal with c source spread > over multiple files. I had some success with 'cat *.c rdoc_zoom.c' > then 'rdoc1.8 rdoc_zoom.c'. But this included all methods under > ZOOM::ResultSet even if they belonged to a different class. > > Here's another approach to get rdoc to work that I haven't attempted: > http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/244008 > > The other option would be to change the rdtool script that's in the > www directory. I didn't like how the site is organized now (preferring > rdoc), but updating what we've got now would be better than nothing. > > How should we proceed with documentation? > > --Jason > > > > On 9/10/07, Walter McGinnis wrote: >> What sort of process do we now want to fulfill before releasing the >> next version of the gem? Have we covered all the major questions, >> etc? From jronallo at gmail.com Wed Sep 12 08:57:16 2007 From: jronallo at gmail.com (Jason Ronallo) Date: Wed, 12 Sep 2007 08:57:16 -0400 Subject: [Ruby-zoom-devel] Where are we in relation to a release In-Reply-To: <1CFDA9D9-81C6-4F6B-86DA-8BB19AED9F40@katipo.co.nz> References: <763570460708270725i3ab904fcy6c368b8e7398be80@mail.gmail.com> <460C1697-03E5-4A64-A91B-5AD2A4382C45@gmail.com> <763570460708280614l6ea37d1aw9c1e7d72121650d5@mail.gmail.com> <3E7F8F50-7328-4381-87EC-EF55C6ED4990@gmail.com> <763570460709060540s153178a9s37fed461fe94291e@mail.gmail.com> <1CFDA9D9-81C6-4F6B-86DA-8BB19AED9F40@katipo.co.nz> Message-ID: <763570460709120557n325bab43t911d8dd9ceee5e6a@mail.gmail.com> Yeah, for the list now. :) First, the 0.4.0 version is currently available from rubyforge though it doesn't show up on the rubyforge webpage. Since at least one bug fix has been submitted since that release, I'd like to see the next release as 0.4.1. I'd like to see documentation and update of the website, at least the addition of extended services. I tried writing an rdoc rake task but kept getting errors about not knowing the enclosing module. This seems to be a problem with rdoc not knowing how to deal with c source spread over multiple files. I had some success with 'cat *.c rdoc_zoom.c' then 'rdoc1.8 rdoc_zoom.c'. But this included all methods under ZOOM::ResultSet even if they belonged to a different class. Here's another approach to get rdoc to work that I haven't attempted: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/244008 The other option would be to change the rdtool script that's in the www directory. I didn't like how the site is organized now (preferring rdoc), but updating what we've got now would be better than nothing. How should we proceed with documentation? Update: I'm going to see if we can still use lrz's extdoc, at least for now. I'll let you know how that goes. --Jason On 9/10/07, Walter McGinnis wrote: > What sort of process do we now want to fulfill before releasing the > next version of the gem? Have we covered all the major questions, etc? > > Cheers, > Walter > > _______________________________________________ > Ruby-zoom-devel mailing list > Ruby-zoom-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/ruby-zoom-devel > From terry.reese at oregonstate.edu Thu Sep 13 14:31:44 2007 From: terry.reese at oregonstate.edu (Reese, Terry) Date: Thu, 13 Sep 2007 11:31:44 -0700 Subject: [Ruby-zoom-devel] Adding Batch record retrieval back into ruby-zoom Message-ID: <1454CCD2A8D4B74F9EEE8CB68C891CC901D6C7B4@NWS-EXCH3.nws.oregonstate.edu> Hi folks, Ed told me I should send this to the development list. I'm currently using the ruby-zoom module in our development of LibraryFind (a metasearch tool) and one of the complains that I've been getting from vendors is the high number of record/search request ratio, which can sometimes be as high as 25:1 (25 record request per search). I'd kindof assumed that the ruby-zoom utilized yaz's get_records command and noticed that it did up until revision 36 of the rbzoomresultset.c file. In the description, it notes that the batch request was removed in favor of a record by record request due to some unexpected results when dealing with some servers. I'd like to propose that you bring the batch retrieval back into the code, maybe including record by record as a fall back. Simply test the results and proceed accordingly. For our purposes, I've made the following changes to the rbz_resultset_index function in the rbzoomresultset.c file (changes are bolded -- see below). Now, I haven't found a server where the batch retrieval method doesn't work so this might need to be tweaked slightly, but I wanted to push this to the list since I'm hoping to not have to maintain our own fork of the code within our project. Thanks, --TR ******************************************* Terry Reese Cataloger for Networked Resources Digital Production Unit Head Oregon State University Libraries Corvallis, OR 97331 tel: 541-737-6384 email: terry.reese at oregonstate.edu http: http://oregonstate.edu/~reeset ******************************************* rbz_resultset_index (int argc, VALUE *argv, VALUE self) { ZOOM_record *records; ZOOM_record record; VALUE ary; size_t begin; size_t count; size_t i; if (argc == 1) { VALUE arg = argv [0]; if (TYPE (arg) == T_FIXNUM || TYPE (arg) == T_BIGNUM) { record = ZOOM_resultset_record (rbz_resultset_get (self), NUM2LONG (arg)); return record != NULL ? rbz_record_make (ZOOM_record_clone (record)) : Qnil; } if (CLASS_OF (arg) == rb_cRange) { begin = NUM2LONG (rb_funcall (arg, rb_intern ("begin"), 0)); count = NUM2LONG (rb_funcall (arg, rb_intern ("end"), 0)); count -= begin; } else rb_raise (rb_eArgError, "Invalid argument of type %s (not Numeric or Range)", rb_class2name (CLASS_OF (arg))); } else { VALUE rb_begin; VALUE rb_count; rb_scan_args (argc, argv, "2", &rb_begin, &rb_count); begin = NUM2LONG (rb_begin); count = NUM2LONG (rb_count); } ary = rb_ary_new (); if (count == 0) return ary; records = ALLOC_N (ZOOM_record, count); /* attempt to retrieve records in batch. */ ZOOM_resultset_records (rbz_resultset_get (self), records, begin, count); /* If the first record in the array is null, fall back to record by record if (records[0]!=NULL) { for (i = 0; i < count; i++) if (records[i]!=NULL) rb_ary_push (ary, rbz_record_make (ZOOM_record_clone (records [i]))); } else { for (i = 0; i < count; i++) { record = ZOOM_resultset_record (rbz_resultset_get (self), begin + i); if (record != NULL) rb_ary_push (ary, rbz_record_make (ZOOM_record_clone (record))); } } return ary; } -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/ruby-zoom-devel/attachments/20070913/2b050a8e/attachment.html From jronallo at gmail.com Thu Sep 13 15:58:53 2007 From: jronallo at gmail.com (Jason Ronallo) Date: Thu, 13 Sep 2007 15:58:53 -0400 Subject: [Ruby-zoom-devel] documentation update Message-ID: <763570460709131258q4b37fd59n91cacecfae995c91@mail.gmail.com> Just wanted to let folks know that I've finally hit upon a way to wring rdoc out of the code. I've written a rake task and made changes to the comments in the source. MAKING_RDOC_HAPPY in each of the classes was the key. Do 'rake rdoc' and take a look. Can we just make the rdoc the homepage for the project and have the README be the first page? Anything else we want for docs before the next release? --Jason From ehs at pobox.com Fri Sep 14 08:40:20 2007 From: ehs at pobox.com (Ed Summers) Date: Fri, 14 Sep 2007 08:40:20 -0400 Subject: [Ruby-zoom-devel] Adding Batch record retrieval back into ruby-zoom In-Reply-To: <1454CCD2A8D4B74F9EEE8CB68C891CC901D6C7B4@NWS-EXCH3.nws.oregonstate.edu> References: <1454CCD2A8D4B74F9EEE8CB68C891CC901D6C7B4@NWS-EXCH3.nws.oregonstate.edu> Message-ID: Hey Terry, thanks for writing. I added you as a developer on rubyforge.org. Could you commit your changes on a branch and let us take them for a test drive? Can you add some unittests? //Ed From terry.reese at oregonstate.edu Fri Sep 14 09:10:14 2007 From: terry.reese at oregonstate.edu (Reese, Terry) Date: Fri, 14 Sep 2007 06:10:14 -0700 Subject: [Ruby-zoom-devel] Adding Batch record retrieval back intoruby-zoom References: <1454CCD2A8D4B74F9EEE8CB68C891CC901D6C7B4@NWS-EXCH3.nws.oregonstate.edu> Message-ID: <1454CCD2A8D4B74F9EEE8CB68C891CC9015D55C3@NWS-EXCH3.nws.oregonstate.edu> Sure things. I'll do it first thing when I get into the office. --TR ******************************************* Terry Reese Cataloger for Networked Resources Digital Production Unit Head Oregon State University Libraries Corvallis, OR 97331 tel: 541-737-6384 email: terry.reese at oregonstate.edu http: http://oregonstate.edu/~reeset ******************************************* ________________________________ From: ruby-zoom-devel-bounces at rubyforge.org on behalf of Ed Summers Sent: Fri 9/14/2007 5:40 AM To: ruby-zoom-devel at rubyforge.org Subject: Re: [Ruby-zoom-devel] Adding Batch record retrieval back intoruby-zoom Hey Terry, thanks for writing. I added you as a developer on rubyforge.org. Could you commit your changes on a branch and let us take them for a test drive? Can you add some unittests? //Ed _______________________________________________ Ruby-zoom-devel mailing list Ruby-zoom-devel at rubyforge.org http://rubyforge.org/mailman/listinfo/ruby-zoom-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/ruby-zoom-devel/attachments/20070914/3121b20f/attachment.html From terry.reese at oregonstate.edu Mon Sep 17 12:36:43 2007 From: terry.reese at oregonstate.edu (Reese, Terry) Date: Mon, 17 Sep 2007 09:36:43 -0700 Subject: [Ruby-zoom-devel] Batch_download branch Message-ID: <1454CCD2A8D4B74F9EEE8CB68C891CC901D6CC1B@NWS-EXCH3.nws.oregonstate.edu> Et all, I created a new branch and reimplemented the get_records function in yaz for batch record retrieval with a fall back for servers that this doesn't seem to work with (http://ruby-zoom.rubyforge.org/svn/branches/batch_download/). BTW, if Laurent still knows of a server or two where the batch record requesting didn't work -- that would be great for testing. I've tried all most of our vendor sites and it seems to work like a treat. --TR -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/ruby-zoom-devel/attachments/20070917/f5a91682/attachment-0001.html From jronallo at gmail.com Tue Sep 18 23:46:20 2007 From: jronallo at gmail.com (Jason Ronallo) Date: Tue, 18 Sep 2007 23:46:20 -0400 Subject: [Ruby-zoom-devel] libyaz3_3.0.10 and segfault Message-ID: <763570460709182046h594a6723yf6e88b3aa5577b1@mail.gmail.com> Hi, The latest libyaz3/libyaz3-dev out of the indexdata repo cause a segmentation fault when trying to retrieve a record out of a result set. I tried the following on two different systems. Any ideas? --Jason I upgraded my Debian system to the latest libyaz3 and libyaz3-dev (3.0.12) out of the indexdata repo, rebuilt the gem and installed it. irb(main):001:0> require 'rubygems' => true irb(main):002:0> require 'zoom' => true irb(main):003:0> c = ZOOM::Connection.open('z3950.loc.gov:7090/Voyager') => # irb(main):004:0> result_set = c.search("@attr 1=4 \"cold beer\"") => # irb(main):005:0> result_set.length => 18 irb(main):006:0> p result_set[0] (irb):6: [BUG] Segmentation fault ruby 1.8.5 (2006-08-25) [i486-linux] Aborted $ Downgraded libyaz3 and libyaz3-dev 3.0.10 with dpkg, rebuild gem and install. irb(main):001:0> require 'rubygems' => true irb(main):002:0> require 'zoom' => true irb(main):003:0> c = ZOOM::Connection.open('z3950.loc.gov:7090/ => # irb(main):004:0> result_set = c.search("@attr 1=4 \"cold beer\"") => # irb(main):005:0> result_set.length => 18 irb(main):006:0> p result_set[0] 02574cam 22003254a 4500 001 13867626 005 20060722213436.0 008 050210s2005 enk b 001 0 eng 906 $a 7 $b cbc $c orignew $d 1 $e ecip $f 20 $g y-gencatlg 925 0 $a acquire $b 2 shelf copies $x policy default 955 $a sf03 2005-02-10 $i sf03 2005-02-10 (rev. sf12) $e sf04 2005-02-22 to Dewey $a aa20 2005-02-22 $a ps05 2005-08-30 1 copy rec'd., to CIP ver. $f sf10 2005-09-21 $g sf10 2005-09-21 to BCCD $a ld11 2006-05-08 copy 2 added 010 $a 2005003348 020 $a 1859738737 (pbk.) 020 $a 1859738680 (cloth) 040 $a DLC $c DLC $d DLC 042 $a pcc 050 00 $a GT2884 $b .D75 2005 082 00 $a 394.1/2 $2 22 245 00 $a Drinking cultures : $b alcohol and identity / $c edited by Thomas M. Wilson. 260 $a Oxford ; $a New York : $b Berg, $c 2005. 300 $a xv, 281 p. ; $c 24 cm. 504 $a Includes bibliographical references and index. 505 0 $a Drinking cultures : sites and practices in the production and expression of identity / Thomas M. Wilson -- Drinking country : flows of exchange in a Japanese valley / Brian Moeran -- Cold beer, warm hearts : community, belonging, and desire in Irish pubs in Berlin / Cliona O'Carroll -- Pivo at the heart of Europe : beer drinking and Czech identities / Timothy M. Hall -- Drunk and (dis)orderly : Norwegian drinking parties in the home / Pauline Garvey -- Cognac, beer, red wine, or soft drinks? : Hong Kong identity and wedding banquets / Josephine Smart -- Consuming wine in France : the "wandering" drinker and the vin-anomie / Marion Demossier -- Romantic moods : food, beer, music, and the Yucatecan soul / Steffan Igor Ayora-Diaz and Gabriela Vargas-Cetina -- Cheers and booze : football and festa drinking in Malta / Jon P. Mitchell and Gary Armstrong -- Drinking rituals, identity, and politics in a Basque town / Sharryn Kasmir -- Alcohol and masculinity : the case of ethnic youth gangs / Geoffrey P. Hunt, Kathleen MacKenzie, and Karen Joe-Laidler -- Drinking politics : alcohol, drugs, and the problem of US civil society / Anthony Marcus. 650 0 $a Drinking of alcoholic beverages. 650 0 $a Drinking customs. 650 0 $a Alcoholic beverages $x Social aspects. 700 1 $a Wilson, Thomas M., $d 1951- 856 41 $3 Table of contents only $u http://www.loc.gov/catdir/toc/ecip058/2005003348.html 856 42 $3 Contributor biographical information $u http://www.loc.gov/catdir/enhancements/fy0621/2005003348-b.html 856 42 $3 Publisher description $u http://www.loc.gov/catdir/enhancements/fy0621/2005003348-d.html => nil irb(main):007:0> From walter at katipo.co.nz Wed Sep 19 00:05:59 2007 From: walter at katipo.co.nz (Walter McGinnis) Date: Wed, 19 Sep 2007 16:05:59 +1200 Subject: [Ruby-zoom-devel] libyaz3_3.0.10 and segfault In-Reply-To: <763570460709182046h594a6723yf6e88b3aa5577b1@mail.gmail.com> References: <763570460709182046h594a6723yf6e88b3aa5577b1@mail.gmail.com> Message-ID: Yep, I have experienced the same thing. Very frustrating. I've been meaning to question the YAZ mailinglist about it, but haven't had the time yet. I actually mention this as a problem in my installation guides for Kete: http://kete.net.nz/documentation/topics/show/16-creating-a-kete- development-environment-on-mac-os-x and http://kete.net.nz/documentation/topics/show/15-preparing-a-debian- etch-host-to-be-deployed-to-for-kete Urgh, I just noticed that TinyMCE has hosed my code formatting! Sorry about that. Another piece of third party code to monkey with. Cheers, Walter From jronallo at gmail.com Wed Sep 19 12:16:20 2007 From: jronallo at gmail.com (Jason Ronallo) Date: Wed, 19 Sep 2007 12:16:20 -0400 Subject: [Ruby-zoom-devel] libyaz3_3.0.10 and segfault In-Reply-To: <763570460709182046h594a6723yf6e88b3aa5577b1@mail.gmail.com> References: <763570460709182046h594a6723yf6e88b3aa5577b1@mail.gmail.com> Message-ID: <763570460709190916u1e07143ax23a11a95ce78de97@mail.gmail.com> I've taken a closer look at this now and it seems to be an issue with threads. I installed the latest yaz packages from indexdata from source on a 64bit system. If I run the script below I get results in the first instance and a seg fault in the second. The only difference is that one is put in a thread. >From my irb session (far below) I saw that it was probably a problem in rbzoomresultset.c. I placed puts throughout rbz_resultset_index. The second one seems to choke on this line towards the end: rb_ary_push (ary, rbz_record_make (ZOOM_record_clone (record))); It gets within the if directly above this line but does not get past the rb_ary_push line. I tried the same little test script with the batch_download branch and got the same results. And here I've reached the limit of what I know to do or try. Any ideas? --Jason require 'rubygems' require 'zoom' ZOOM::Connection.open('z3950.loc.gov:7090/Voyager') do |conn| result_set = conn.search('gatsby') puts result_set[0, 2] end puts "hit enter to continue to second batch" STDIN.gets t = Thread.new do ZOOM::Connection.open('z3950.loc.gov:7090/Voyager') do |c| rs = c.search('gatsby') puts rs[0, 2] end end t.join On 9/18/07, Jason Ronallo wrote: > Hi, > > The latest libyaz3/libyaz3-dev out of the indexdata repo cause a > segmentation fault when trying to retrieve a record out of a result > set. I tried the following on two different systems. Any ideas? > > --Jason > > I upgraded my Debian system to the latest libyaz3 and libyaz3-dev > (3.0.12) out of the indexdata repo, rebuilt the gem and installed it. > > irb(main):001:0> require 'rubygems' > => true > irb(main):002:0> require 'zoom' > => true > irb(main):003:0> c = ZOOM::Connection.open('z3950.loc.gov:7090/Voyager') > => # > irb(main):004:0> result_set = c.search("@attr 1=4 \"cold beer\"") > => # > irb(main):005:0> result_set.length > => 18 > irb(main):006:0> p result_set[0] > (irb):6: [BUG] Segmentation fault > ruby 1.8.5 (2006-08-25) [i486-linux] > > Aborted > $ > > Downgraded libyaz3 and libyaz3-dev 3.0.10 with dpkg, rebuild gem and install. > > irb(main):001:0> require 'rubygems' > => true > irb(main):002:0> require 'zoom' > => true > irb(main):003:0> c = ZOOM::Connection.open('z3950.loc.gov:7090/ > => # > irb(main):004:0> result_set = c.search("@attr 1=4 \"cold beer\"") > => # > irb(main):005:0> result_set.length > => 18 > irb(main):006:0> p result_set[0] > 02574cam 22003254a 4500 > 001 13867626 > 005 20060722213436.0 > 008 050210s2005 enk b 001 0 eng > 906 $a 7 $b cbc $c orignew $d 1 $e ecip $f 20 $g y-gencatlg > 925 0 $a acquire $b 2 shelf copies $x policy default > 955 $a sf03 2005-02-10 $i sf03 2005-02-10 (rev. sf12) $e sf04 > 2005-02-22 to Dewey $a aa20 2005-02-22 $a ps05 2005-08-30 1 copy > rec'd., to CIP ver. $f sf10 2005-09-21 $g sf10 2005-09-21 to BCCD $a > ld11 2006-05-08 copy 2 added > 010 $a 2005003348 > 020 $a 1859738737 (pbk.) > 020 $a 1859738680 (cloth) > 040 $a DLC $c DLC $d DLC > 042 $a pcc > 050 00 $a GT2884 $b .D75 2005 > 082 00 $a 394.1/2 $2 22 > 245 00 $a Drinking cultures : $b alcohol and identity / $c edited by > Thomas M. Wilson. > 260 $a Oxford ; $a New York : $b Berg, $c 2005. > 300 $a xv, 281 p. ; $c 24 cm. > 504 $a Includes bibliographical references and index. > 505 0 $a Drinking cultures : sites and practices in the production > and expression of identity / Thomas M. Wilson -- Drinking country : > flows of exchange in a Japanese valley / Brian Moeran -- Cold beer, > warm hearts : community, belonging, and desire in Irish pubs in Berlin > / Cliona O'Carroll -- Pivo at the heart of Europe : beer drinking and > Czech identities / Timothy M. Hall -- Drunk and (dis)orderly : > Norwegian drinking parties in the home / Pauline Garvey -- Cognac, > beer, red wine, or soft drinks? : Hong Kong identity and wedding > banquets / Josephine Smart -- Consuming wine in France : the > "wandering" drinker and the vin-anomie / Marion Demossier -- Romantic > moods : food, beer, music, and the Yucatecan soul / Steffan Igor > Ayora-Diaz and Gabriela Vargas-Cetina -- Cheers and booze : football > and festa drinking in Malta / Jon P. Mitchell and Gary Armstrong -- > Drinking rituals, identity, and politics in a Basque town / Sharryn > Kasmir -- Alcohol and masculinity : the case of ethnic youth gangs / > Geoffrey P. Hunt, Kathleen MacKenzie, and Karen Joe-Laidler -- > Drinking politics : alcohol, drugs, and the problem of US civil > society / Anthony Marcus. > 650 0 $a Drinking of alcoholic beverages. > 650 0 $a Drinking customs. > 650 0 $a Alcoholic beverages $x Social aspects. > 700 1 $a Wilson, Thomas M., $d 1951- > 856 41 $3 Table of contents only $u > http://www.loc.gov/catdir/toc/ecip058/2005003348.html > 856 42 $3 Contributor biographical information $u > http://www.loc.gov/catdir/enhancements/fy0621/2005003348-b.html > 856 42 $3 Publisher description $u > http://www.loc.gov/catdir/enhancements/fy0621/2005003348-d.html > > > => nil > irb(main):007:0> > From jronallo at gmail.com Wed Sep 19 14:14:28 2007 From: jronallo at gmail.com (Jason Ronallo) Date: Wed, 19 Sep 2007 14:14:28 -0400 Subject: [Ruby-zoom-devel] libyaz3_3.0.10 and segfault In-Reply-To: <763570460709190916u1e07143ax23a11a95ce78de97@mail.gmail.com> References: <763570460709182046h594a6723yf6e88b3aa5577b1@mail.gmail.com> <763570460709190916u1e07143ax23a11a95ce78de97@mail.gmail.com> Message-ID: <763570460709191114j3ab63c07u962966e50a478d18@mail.gmail.com> At least to me, it gets even stranger. If I run my test script with the thread using the gem then it seg faults. If I uninstall the gem and have zoom.so in the same directory as my script then it runs just fine. Any ideas now? --Jason On 9/19/07, Jason Ronallo wrote: > I've taken a closer look at this now and it seems to be an issue with > threads. I installed the latest yaz packages from indexdata from > source on a 64bit system. > > If I run the script below I get results in the first instance and a > seg fault in the second. The only difference is that one is put in a > thread. > > From my irb session (far below) I saw that it was probably a problem > in rbzoomresultset.c. I placed puts throughout rbz_resultset_index. > The second one seems to choke on this line towards the end: > > rb_ary_push (ary, rbz_record_make (ZOOM_record_clone (record))); > > It gets within the if directly above this line but does not get past > the rb_ary_push line. > > I tried the same little test script with the batch_download branch and > got the same results. > > And here I've reached the limit of what I know to do or try. Any ideas? > > --Jason > > require 'rubygems' > require 'zoom' > > ZOOM::Connection.open('z3950.loc.gov:7090/Voyager') do |conn| > result_set = conn.search('gatsby') > puts result_set[0, 2] > end > > puts "hit enter to continue to second batch" > STDIN.gets > > t = Thread.new do > ZOOM::Connection.open('z3950.loc.gov:7090/Voyager') do |c| > rs = c.search('gatsby') > puts rs[0, 2] > end > end > > t.join > > > On 9/18/07, Jason Ronallo wrote: > > Hi, > > > > The latest libyaz3/libyaz3-dev out of the indexdata repo cause a > > segmentation fault when trying to retrieve a record out of a result > > set. I tried the following on two different systems. Any ideas? > > > > --Jason > > > > I upgraded my Debian system to the latest libyaz3 and libyaz3-dev > > (3.0.12) out of the indexdata repo, rebuilt the gem and installed it. > > > > irb(main):001:0> require 'rubygems' > > => true > > irb(main):002:0> require 'zoom' > > => true > > irb(main):003:0> c = ZOOM::Connection.open('z3950.loc.gov:7090/Voyager') > > => # > > irb(main):004:0> result_set = c.search("@attr 1=4 \"cold beer\"") > > => # > > irb(main):005:0> result_set.length > > => 18 > > irb(main):006:0> p result_set[0] > > (irb):6: [BUG] Segmentation fault > > ruby 1.8.5 (2006-08-25) [i486-linux] > > > > Aborted > > $ > > > > Downgraded libyaz3 and libyaz3-dev 3.0.10 with dpkg, rebuild gem and install. > > > > irb(main):001:0> require 'rubygems' > > => true > > irb(main):002:0> require 'zoom' > > => true > > irb(main):003:0> c = ZOOM::Connection.open('z3950.loc.gov:7090/ > > => # > > irb(main):004:0> result_set = c.search("@attr 1=4 \"cold beer\"") > > => # > > irb(main):005:0> result_set.length > > => 18 > > irb(main):006:0> p result_set[0] > > 02574cam 22003254a 4500 > > 001 13867626 > > 005 20060722213436.0 > > 008 050210s2005 enk b 001 0 eng > > 906 $a 7 $b cbc $c orignew $d 1 $e ecip $f 20 $g y-gencatlg > > 925 0 $a acquire $b 2 shelf copies $x policy default > > 955 $a sf03 2005-02-10 $i sf03 2005-02-10 (rev. sf12) $e sf04 > > 2005-02-22 to Dewey $a aa20 2005-02-22 $a ps05 2005-08-30 1 copy > > rec'd., to CIP ver. $f sf10 2005-09-21 $g sf10 2005-09-21 to BCCD $a > > ld11 2006-05-08 copy 2 added > > 010 $a 2005003348 > > 020 $a 1859738737 (pbk.) > > 020 $a 1859738680 (cloth) > > 040 $a DLC $c DLC $d DLC > > 042 $a pcc > > 050 00 $a GT2884 $b .D75 2005 > > 082 00 $a 394.1/2 $2 22 > > 245 00 $a Drinking cultures : $b alcohol and identity / $c edited by > > Thomas M. Wilson. > > 260 $a Oxford ; $a New York : $b Berg, $c 2005. > > 300 $a xv, 281 p. ; $c 24 cm. > > 504 $a Includes bibliographical references and index. > > 505 0 $a Drinking cultures : sites and practices in the production > > and expression of identity / Thomas M. Wilson -- Drinking country : > > flows of exchange in a Japanese valley / Brian Moeran -- Cold beer, > > warm hearts : community, belonging, and desire in Irish pubs in Berlin > > / Cliona O'Carroll -- Pivo at the heart of Europe : beer drinking and > > Czech identities / Timothy M. Hall -- Drunk and (dis)orderly : > > Norwegian drinking parties in the home / Pauline Garvey -- Cognac, > > beer, red wine, or soft drinks? : Hong Kong identity and wedding > > banquets / Josephine Smart -- Consuming wine in France : the > > "wandering" drinker and the vin-anomie / Marion Demossier -- Romantic > > moods : food, beer, music, and the Yucatecan soul / Steffan Igor > > Ayora-Diaz and Gabriela Vargas-Cetina -- Cheers and booze : football > > and festa drinking in Malta / Jon P. Mitchell and Gary Armstrong -- > > Drinking rituals, identity, and politics in a Basque town / Sharryn > > Kasmir -- Alcohol and masculinity : the case of ethnic youth gangs / > > Geoffrey P. Hunt, Kathleen MacKenzie, and Karen Joe-Laidler -- > > Drinking politics : alcohol, drugs, and the problem of US civil > > society / Anthony Marcus. > > 650 0 $a Drinking of alcoholic beverages. > > 650 0 $a Drinking customs. > > 650 0 $a Alcoholic beverages $x Social aspects. > > 700 1 $a Wilson, Thomas M., $d 1951- > > 856 41 $3 Table of contents only $u > > http://www.loc.gov/catdir/toc/ecip058/2005003348.html > > 856 42 $3 Contributor biographical information $u > > http://www.loc.gov/catdir/enhancements/fy0621/2005003348-b.html > > 856 42 $3 Publisher description $u > > http://www.loc.gov/catdir/enhancements/fy0621/2005003348-d.html > > > > > > => nil > > irb(main):007:0> > > > From jronallo at gmail.com Thu Sep 20 11:12:41 2007 From: jronallo at gmail.com (Jason Ronallo) Date: Thu, 20 Sep 2007 11:12:41 -0400 Subject: [Ruby-zoom-devel] thread_test for yaz-3.0.12 Message-ID: <763570460709200812s2f8069cbja2296ca8e7e67c42@mail.gmail.com> Continuing with the problems with yaz-3.0.12. I added search_batch_test.rb from branch/batch_download to trunk. This test tries to retrieve 10 records out of a resultset. It passes for me. I also added test_thread.rb and changed the Rakefile to add task 'rake thread_test'. This test tries to retrieve 6 records from a resultset. All the connection creation, search, and retrieval of the records happens within a thread. thread_test fails for me using yaz-3.0.12 and passes with yaz-3.0.10. It passed the test once using yaz-3.0.12, if I change the number to only retrieve 5 records out of the resultset. But subsequently even <5 records failed for me as well. If this test passes for you using yaz-3.0.12 could you please change the test to use a higher number? --Jason From walter at katipo.co.nz Thu Sep 20 18:02:00 2007 From: walter at katipo.co.nz (Walter McGinnis) Date: Fri, 21 Sep 2007 10:02:00 +1200 Subject: [Ruby-zoom-devel] thread_test for yaz-3.0.12 In-Reply-To: <763570460709200812s2f8069cbja2296ca8e7e67c42@mail.gmail.com> References: <763570460709200812s2f8069cbja2296ca8e7e67c42@mail.gmail.com> Message-ID: I really think it's a good idea to bring this to the attention of the YAZ team over at indexdata. I would do it myself, but you have a much more complete understanding of where the problems are. Here's their mailinglist info (which looks like their main line of communication with the community): http://lists.indexdata.dk/cgi-bin/mailman/listinfo/yazlist Cheers, Walter On Sep 21, 2007, at 3:12 AM, Jason Ronallo wrote: > Continuing with the problems with yaz-3.0.12. > > I added search_batch_test.rb from branch/batch_download to trunk. > This test tries to retrieve 10 records out of a resultset. It > passes for me. > > I also added test_thread.rb and changed the Rakefile to add task 'rake > thread_test'. This test tries to retrieve 6 records from a resultset. > All the connection creation, search, and retrieval of the records > happens within a thread. > > thread_test fails for me using yaz-3.0.12 and passes with yaz-3.0.10. > It passed the test once using yaz-3.0.12, if I change the number to > only retrieve 5 records out of the resultset. But subsequently even <5 > records failed for me as well. If this test passes for you using > yaz-3.0.12 could you please change the test to use a higher number? > > --Jason > _______________________________________________ > Ruby-zoom-devel mailing list > Ruby-zoom-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/ruby-zoom-devel From jronallo at gmail.com Thu Sep 20 18:53:59 2007 From: jronallo at gmail.com (Jason Ronallo) Date: Thu, 20 Sep 2007 18:53:59 -0400 Subject: [Ruby-zoom-devel] thread_test for yaz-3.0.12 In-Reply-To: References: <763570460709200812s2f8069cbja2296ca8e7e67c42@mail.gmail.com> Message-ID: <763570460709201553p3d82a3b8m1828fc2178495f23@mail.gmail.com> I wrote them today. You can follow the thread here: http://lists.indexdata.dk/pipermail/yazlist/2007-September/002226.html Mike Taylor wrote:"Best of all would be to translate your failing Ruby program into an equivalent ZOOM-C program -- that way we don't have to deal with the upper layers in trying to debug it." But unfortunately I'm not capable of that. :( If it's possible, please run the unit tests and let me know your results. I'm still not certain that it doesn't have to do with system configuration or my local dynamic linking of run time bindings--which I'm not sure exactly what that means yet. --Jason On 9/20/07, Walter McGinnis wrote: > I really think it's a good idea to bring this to the attention of the > YAZ team over at indexdata. I would do it myself, but you have a > much more complete understanding of where the problems are. > > Here's their mailinglist info (which looks like their main line of > communication with the community): > > http://lists.indexdata.dk/cgi-bin/mailman/listinfo/yazlist > > Cheers, > Walter > > On Sep 21, 2007, at 3:12 AM, Jason Ronallo wrote: > > > Continuing with the problems with yaz-3.0.12. > > > > I added search_batch_test.rb from branch/batch_download to trunk. > > This test tries to retrieve 10 records out of a resultset. It > > passes for me. > > > > I also added test_thread.rb and changed the Rakefile to add task 'rake > > thread_test'. This test tries to retrieve 6 records from a resultset. > > All the connection creation, search, and retrieval of the records > > happens within a thread. > > > > thread_test fails for me using yaz-3.0.12 and passes with yaz-3.0.10. > > It passed the test once using yaz-3.0.12, if I change the number to > > only retrieve 5 records out of the resultset. But subsequently even <5 > > records failed for me as well. If this test passes for you using > > yaz-3.0.12 could you please change the test to use a higher number? > > > > --Jason > > _______________________________________________ > > Ruby-zoom-devel mailing list > > Ruby-zoom-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/ruby-zoom-devel > > _______________________________________________ > Ruby-zoom-devel mailing list > Ruby-zoom-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/ruby-zoom-devel > From jronallo at gmail.com Fri Sep 21 08:59:44 2007 From: jronallo at gmail.com (Jason Ronallo) Date: Fri, 21 Sep 2007 08:59:44 -0400 Subject: [Ruby-zoom-devel] yaz 3.0.14 fixes segfault Message-ID: <763570460709210559s763a4050t9b8a01a0758c0623@mail.gmail.com> Hi, Good news. Indexdata was able to repeat the problems with yaz-3.0.12 and fix it. Changes for yaz-3.0.14 include: "Fixed bad memory reference in ZOOM_record - cuased by member not being initialized. This was a problem only in YAZ 3.0.12." The thread_test now passes, so I've moved it to be run with the rest of the default tests. I added the important changes since last update to the ChangeLog in preparation of releasing a new version of the gem (0.4.1?). Anything else need to happen to make a release? --Jason From jronallo at gmail.com Thu Sep 27 11:55:53 2007 From: jronallo at gmail.com (Jason Ronallo) Date: Thu, 27 Sep 2007 11:55:53 -0400 Subject: [Ruby-zoom-devel] merged batch_download branch into trunk Message-ID: <763570460709270855h30033b38o7f3d1385eca344fc@mail.gmail.com> Hi, Meant to send this earlier, but schoolwork intervened. In consultation with Ed, I merged the batch_download branch into trunk. I had emailed Laurent asking if he had any record or recollection of what targets had exhibited the problem but haven't received an answer. In my testing I can retrieve records from all targets I regularly use, and Terry reports that it works for all their vendors. I figure if it works for us and we don't have a test case target like Laurent noticed where the batch download failed, let's put it out there. I'm confident Terry will help fix it if we do get any reports or problems. ;) --Jason From terry.reese at oregonstate.edu Thu Sep 27 13:48:54 2007 From: terry.reese at oregonstate.edu (Reese, Terry) Date: Thu, 27 Sep 2007 10:48:54 -0700 Subject: [Ruby-zoom-devel] merged batch_download branch into trunk References: <763570460709270855h30033b38o7f3d1385eca344fc@mail.gmail.com> Message-ID: <1454CCD2A8D4B74F9EEE8CB68C891CC9015D563E@NWS-EXCH3.nws.oregonstate.edu> >> I'm confident Terry will help fix it if we do get any reports or problems. ;) Of course. ******************************************* Terry Reese Cataloger for Networked Resources Digital Production Unit Head Oregon State University Libraries Corvallis, OR 97331 tel: 541-737-6384 email: terry.reese at oregonstate.edu http: http://oregonstate.edu/~reeset ******************************************* ________________________________ From: Jason Ronallo [mailto:jronallo at gmail.com] Sent: Thu 9/27/2007 8:55 AM To: Reese, Terry Cc: ruby-zoom-devel at rubyforge.org Subject: merged batch_download branch into trunk Hi, Meant to send this earlier, but schoolwork intervened. In consultation with Ed, I merged the batch_download branch into trunk. I had emailed Laurent asking if he had any record or recollection of what targets had exhibited the problem but haven't received an answer. In my testing I can retrieve records from all targets I regularly use, and Terry reports that it works for all their vendors. I figure if it works for us and we don't have a test case target like Laurent noticed where the batch download failed, let's put it out there. I'm confident Terry will help fix it if we do get any reports or problems. ;) --Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/ruby-zoom-devel/attachments/20070927/d66dbd90/attachment.html