From hammy at charter.net Tue May 1 18:00:22 2007 From: hammy at charter.net (Eric Schultz) Date: Tue, 1 May 2007 17:00:22 -0500 Subject: [libxml-devel] PATCH: Enable Text Node Creation Message-ID: <000001c78c3c$1dca6bc0$595f4340$@net> Hi everyone, I've been working on using the libxml-ruby for a project and I discovered there's not an easy way to create a new text node. Since I needed this, I used my rather limited C skills and copied the method to create a comment node and modified it so it makes text nodes. The patch is attached. (I hope that's alright.) To whomever takes care of adding things to the repository, you can take care of that I. Thanks for all the work that's already been done! Eric Schultz -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/libxml-devel/attachments/20070501/53a48332/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: patch.diff Type: application/octet-stream Size: 1697 bytes Desc: not available Url : http://rubyforge.org/pipermail/libxml-devel/attachments/20070501/53a48332/attachment-0001.obj From jburgess777 at googlemail.com Thu May 24 19:29:23 2007 From: jburgess777 at googlemail.com (Jon Burgess) Date: Fri, 25 May 2007 00:29:23 +0100 Subject: [libxml-devel] [patch] calling xmlFreeProp() for attributes Message-ID: <1180049363.4055.3.camel@localhost.localdomain> I noticed that the current code calls xmlFreeNode on attributes which looks wrong, surely this should be xmlFreeProp? Jon -------------- next part -------------- A non-text attachment was scrubbed... Name: libxml-ruby-attr.patch Type: text/x-patch Size: 468 bytes Desc: not available Url : http://rubyforge.org/pipermail/libxml-devel/attachments/20070525/2658af01/attachment.bin From jburgess777 at googlemail.com Thu May 24 19:59:15 2007 From: jburgess777 at googlemail.com (Jon Burgess) Date: Fri, 25 May 2007 00:59:15 +0100 Subject: [libxml-devel] valgrind detects problem when setting attribute to nil Message-ID: <1180051155.4055.12.camel@localhost.localdomain> When the attached code is run under valgrind I get the following error. ==13906== Invalid read of size 8 ==13906== at 0x6B4E13C: ruby_xml_attr_free (ruby_xml_attr.c:12) ==13906== by 0x3C2524A9CD: rb_gc_call_finalizer_at_exit (gc.c:1928) ==13906== by 0x3C2522FA92: ruby_finalize_1 (eval.c:1548) ==13906== by 0x3C25238BFF: ruby_cleanup (eval.c:1582) ==13906== by 0x3C25238D28: ruby_stop (eval.c:1619) ==13906== by 0x3C252444C3: ruby_run (eval.c:1640) ==13906== by 0x4007E2: main (main.c:48) ==13906== Address 0x6AE4790 is 40 bytes inside a block of size 96 free'd ==13906== at 0x4A055AB: free (vg_replace_malloc.c:233) ==13906== by 0x3C2524A737: ruby_xfree (gc.c:159) ==13906== by 0x3C304507AE: xmlRemoveProp (in /usr/lib64/libxml2.so.2.6.28) ==13906== by 0x6B49C3D: ruby_xml_node_property_set (ruby_xml_node.c:1892) ==13906== by 0x3C25235BAD: rb_call0 (eval.c:5815) ==13906== by 0x3C252360B7: rb_call (eval.c:6062) ==13906== by 0x3C2523DAB3: rb_eval (eval.c:3429) ==13906== by 0x3C2524442A: ruby_exec_internal (eval.c:1608) ==13906== by 0x3C25244474: ruby_exec (eval.c:1628) ==13906== by 0x3C2524449F: ruby_run (eval.c:1638) ==13906== by 0x4007E2: main (main.c:48) ==13906== I think the issue is with ruby_xml_node_property_set() calling xmlRemoveProp() which both removes the attribute from the node and frees it. I think freeing the attribute needs to be conditional on there being no other references. Jon -------------- next part -------------- A non-text attachment was scrubbed... Name: attr.rb Type: application/x-ruby Size: 169 bytes Desc: not available Url : http://rubyforge.org/pipermail/libxml-devel/attachments/20070525/698507db/attachment.bin From tom at compton.nu Sat May 26 05:58:36 2007 From: tom at compton.nu (Tom Hughes) Date: Sat, 26 May 2007 10:58:36 +0100 Subject: [libxml-devel] memory leak Message-ID: Adding a node that is not part of a document as a child of a node that is part of a document leaks a copy of the node. The problem is ruby_xml_node_child_set() will copy the node being added as a child into the document of it's new parent, and then changes its second argument to refer to that copy, but it makes no attempt to update the reference count of (and free if necessary) the original node. In a test program that generates 1000 documents each with 1000 nodes in my process grows to 750Mb in size - after removing this leak it runs in a stable 10Mb footprint. The problem with the fix is that it brings the attribute problems that Jon Burgess recently posted about to the fore as it now becomes much more likely that the node has been freed and we then die referencing uninitialised memory trying to free the attribute that was freed along with the node. Tom -- Tom Hughes (tom at compton.nu) http://www.compton.nu/ From tom at compton.nu Sat May 26 05:50:11 2007 From: tom at compton.nu (Tom Hughes) Date: Sat, 26 May 2007 10:50:11 +0100 Subject: [libxml-devel] [patch] calling xmlFreeProp() for attributes In-Reply-To: <1180049363.4055.3.camel@localhost.localdomain> References: <1180049363.4055.3.camel@localhost.localdomain> Message-ID: In message <1180049363.4055.3.camel at localhost.localdomain> jburgess777 at googlemail.com (Jon Burgess) wrote: > I noticed that the current code calls xmlFreeNode on attributes which > looks wrong, surely this should be xmlFreeProp? I thought it looked odd too, but I think it is more complicated than that as the attribute will normally belong to a node and therefore shouldn't be freed on it's own like that anyway. In fact I'm not sure that code should ever be reachable, as the attribute will always have a parent node I think. Tom -- Tom Hughes (tom at compton.nu) http://www.compton.nu/ From tom at compton.nu Sat May 26 05:52:58 2007 From: tom at compton.nu (Tom Hughes) Date: Sat, 26 May 2007 10:52:58 +0100 Subject: [libxml-devel] valgrind detects problem when setting attribute to nil In-Reply-To: <1180051155.4055.12.camel@localhost.localdomain> References: <1180051155.4055.12.camel@localhost.localdomain> Message-ID: <0bc6fae84e.tom@loxley.compton.nu> In message <1180051155.4055.12.camel at localhost.localdomain> jburgess777 at googlemail.com (Jon Burgess) wrote: > When the attached code is run under valgrind I get the following error. > > ==13906== Invalid read of size 8 > ==13906== at 0x6B4E13C: ruby_xml_attr_free (ruby_xml_attr.c:12) > ==13906== by 0x3C2524A9CD: rb_gc_call_finalizer_at_exit (gc.c:1928) > ==13906== by 0x3C2522FA92: ruby_finalize_1 (eval.c:1548) > ==13906== by 0x3C25238BFF: ruby_cleanup (eval.c:1582) > ==13906== by 0x3C25238D28: ruby_stop (eval.c:1619) > ==13906== by 0x3C252444C3: ruby_run (eval.c:1640) > ==13906== by 0x4007E2: main (main.c:48) > ==13906== Address 0x6AE4790 is 40 bytes inside a block of size 96 free'd > ==13906== at 0x4A055AB: free (vg_replace_malloc.c:233) > ==13906== by 0x3C2524A737: ruby_xfree (gc.c:159) > ==13906== by 0x3C304507AE: xmlRemoveProp (in /usr/lib64/libxml2.so.2.6.28) > ==13906== by 0x6B49C3D: ruby_xml_node_property_set (ruby_xml_node.c:1892) > ==13906== by 0x3C25235BAD: rb_call0 (eval.c:5815) > ==13906== by 0x3C252360B7: rb_call (eval.c:6062) > ==13906== by 0x3C2523DAB3: rb_eval (eval.c:3429) > ==13906== by 0x3C2524442A: ruby_exec_internal (eval.c:1608) > ==13906== by 0x3C25244474: ruby_exec (eval.c:1628) > ==13906== by 0x3C2524449F: ruby_run (eval.c:1638) > ==13906== by 0x4007E2: main (main.c:48) > ==13906== > > I think the issue is with ruby_xml_node_property_set() calling > xmlRemoveProp() which both removes the attribute from the node and frees > it. I think freeing the attribute needs to be conditional on there being > no other references. That is one problem, yes. It is worse than that though, because if the node which the attribute is attached to is freed then the attribute will be freed (even if there are other references to it) and there is no easy way to find out which attributes need to be kept. I guess you would have to walk the entire node tree checking the _private reference count of each attribute and unlinking anything that is shared before freeing the node? It's horrible anyway... Tom -- Tom Hughes (tom at compton.nu) http://www.compton.nu/ From jburgess777 at googlemail.com Sat May 26 12:13:34 2007 From: jburgess777 at googlemail.com (Jon Burgess) Date: Sat, 26 May 2007 17:13:34 +0100 Subject: [libxml-devel] valgrind detects problem when setting attribute to nil In-Reply-To: <0bc6fae84e.tom@loxley.compton.nu> References: <1180051155.4055.12.camel@localhost.localdomain> <0bc6fae84e.tom@loxley.compton.nu> Message-ID: <1180196014.4055.41.camel@localhost.localdomain> On Sat, 2007-05-26 at 10:52 +0100, Tom Hughes wrote: > In message <1180051155.4055.12.camel at localhost.localdomain> > jburgess777 at googlemail.com (Jon Burgess) wrote: > > > When the attached code is run under valgrind I get the following error. > > > > ==13906== Invalid read of size 8 > > ==13906== at 0x6B4E13C: ruby_xml_attr_free (ruby_xml_attr.c:12) > > ==13906== by 0x3C2524A9CD: rb_gc_call_finalizer_at_exit (gc.c:1928) > > ==13906== by 0x3C2522FA92: ruby_finalize_1 (eval.c:1548) > > ==13906== by 0x3C25238BFF: ruby_cleanup (eval.c:1582) > > ==13906== by 0x3C25238D28: ruby_stop (eval.c:1619) > > ==13906== by 0x3C252444C3: ruby_run (eval.c:1640) > > ==13906== by 0x4007E2: main (main.c:48) > > ==13906== Address 0x6AE4790 is 40 bytes inside a block of size 96 free'd > > ==13906== at 0x4A055AB: free (vg_replace_malloc.c:233) > > ==13906== by 0x3C2524A737: ruby_xfree (gc.c:159) > > ==13906== by 0x3C304507AE: xmlRemoveProp (in /usr/lib64/libxml2.so.2.6.28) > > ==13906== by 0x6B49C3D: ruby_xml_node_property_set (ruby_xml_node.c:1892) > > ==13906== by 0x3C25235BAD: rb_call0 (eval.c:5815) > > ==13906== by 0x3C252360B7: rb_call (eval.c:6062) > > ==13906== by 0x3C2523DAB3: rb_eval (eval.c:3429) > > ==13906== by 0x3C2524442A: ruby_exec_internal (eval.c:1608) > > ==13906== by 0x3C25244474: ruby_exec (eval.c:1628) > > ==13906== by 0x3C2524449F: ruby_run (eval.c:1638) > > ==13906== by 0x4007E2: main (main.c:48) > > ==13906== > > > > I think the issue is with ruby_xml_node_property_set() calling > > xmlRemoveProp() which both removes the attribute from the node and frees > > it. I think freeing the attribute needs to be conditional on there being > > no other references. > > That is one problem, yes. > > It is worse than that though, because if the node which the > attribute is attached to is freed then the attribute will be > freed (even if there are other references to it) and there is > no easy way to find out which attributes need to be kept. > > I guess you would have to walk the entire node tree checking > the _private reference count of each attribute and unlinking > anything that is shared before freeing the node? > > It's horrible anyway... > > Tom > I'm only just starting to get my head around the interaction between C and the ruby objects and GC. The patch below seems to be a simple fix but I'm by no means certain this fixes the more general problems. I think it works because the GC implicitly performs the reference checking for the case you mention. Jon -------------- next part -------------- A non-text attachment was scrubbed... Name: ruby-libxml-attr-fix.patch Type: text/x-patch Size: 526 bytes Desc: not available Url : http://rubyforge.org/pipermail/libxml-devel/attachments/20070526/53c90557/attachment.bin From tom at compton.nu Sat May 26 13:45:51 2007 From: tom at compton.nu (Tom Hughes) Date: Sat, 26 May 2007 18:45:51 +0100 Subject: [libxml-devel] valgrind detects problem when setting attribute In-Reply-To: <1180196014.4055.41.camel@localhost.localdomain> References: <1180051155.4055.12.camel@localhost.localdomain> <0bc6fae84e.tom@loxley.compton.nu> <1180196014.4055.41.camel@localhost.localdomain> Message-ID: In message <1180196014.4055.41.camel at localhost.localdomain> Jon Burgess wrote: > On Sat, 2007-05-26 at 10:52 +0100, Tom Hughes wrote: > > > In message <1180051155.4055.12.camel at localhost.localdomain> > > jburgess777 at googlemail.com (Jon Burgess) wrote: > > > > It is worse than that though, because if the node which the > > attribute is attached to is freed then the attribute will be > > freed (even if there are other references to it) and there is > > no easy way to find out which attributes need to be kept. > > > > I guess you would have to walk the entire node tree checking > > the _private reference count of each attribute and unlinking > > anything that is shared before freeing the node? > > I'm only just starting to get my head around the interaction between C > and the ruby objects and GC. The patch below seems to be a simple fix > but I'm by no means certain this fixes the more general problems. > > I think it works because the GC implicitly performs the reference > checking for the case you mention. Nope. As soon as I fixed the memory leak mentioned in my other thread (which is behind at least some of the OpenStreetMap memory leak issue that I assume we're both looking at) I started getting crashes because the node tree was being free leaving orphaned ruby attribute objects that pointed at libxml attribute objects that had been deleted. I'm working on a plan at the moment, but it currently involves reworking the whole basis of how nodes and attributes are managed... Tom -- Tom Hughes (tom at compton.nu) http://www.compton.nu/ From cjbottaro at alumni.cs.utexas.edu Sat May 26 22:36:59 2007 From: cjbottaro at alumni.cs.utexas.edu (Christopher J. Bottaro) Date: Sat, 26 May 2007 21:36:59 -0500 Subject: [libxml-devel] How to copy nodes from one document to another? Message-ID: Hello, How do I do something like this? doc1 = XML::Document.file(...) doc2 = XML::Document.file(...) some_node = doc1.find(...) another_node = doc2.find(...) some_node << another_node That doesn't work. I tried doing: some_node << another_node.copy But it complains that copy requires one argument, which is not in the documentation. Thanks for the help. From jburgess777 at googlemail.com Sun May 27 17:24:34 2007 From: jburgess777 at googlemail.com (Jon Burgess) Date: Sun, 27 May 2007 22:24:34 +0100 Subject: [libxml-devel] valgrind detects problem when setting attribute In-Reply-To: References: <1180051155.4055.12.camel@localhost.localdomain> <0bc6fae84e.tom@loxley.compton.nu> <1180196014.4055.41.camel@localhost.localdomain> Message-ID: <1180301074.4055.82.camel@localhost.localdomain> On Sat, 2007-05-26 at 18:45 +0100, Tom Hughes wrote: > In message <1180196014.4055.41.camel at localhost.localdomain> > Jon Burgess wrote: > > > On Sat, 2007-05-26 at 10:52 +0100, Tom Hughes wrote: > > > > > In message <1180051155.4055.12.camel at localhost.localdomain> > > > jburgess777 at googlemail.com (Jon Burgess) wrote: > > > > > > It is worse than that though, because if the node which the > > > attribute is attached to is freed then the attribute will be > > > freed (even if there are other references to it) and there is > > > no easy way to find out which attributes need to be kept. > > > > > > I guess you would have to walk the entire node tree checking > > > the _private reference count of each attribute and unlinking > > > anything that is shared before freeing the node? > > > > I'm only just starting to get my head around the interaction between C > > and the ruby objects and GC. The patch below seems to be a simple fix > > but I'm by no means certain this fixes the more general problems. > > > > I think it works because the GC implicitly performs the reference > > checking for the case you mention. > > Nope. As soon as I fixed the memory leak mentioned in my other > thread (which is behind at least some of the OpenStreetMap memory > leak issue that I assume we're both looking at) I started getting > crashes because the node tree was being free leaving orphaned > ruby attribute objects that pointed at libxml attribute objects > that had been deleted. Yes, I'm looking at it from the OSM angle too. Would you mind sharing your initial fix and test case? > I'm working on a plan at the moment, but it currently involves > reworking the whole basis of how nodes and attributes are managed... I'm working on which sounds similar. It is tricky to get libxml and the GC working together successfully. Jon