From cfis at savagexi.com  Mon Jul  9 15:25:48 2007
From: cfis at savagexi.com (Charlie Savage)
Date: Mon, 09 Jul 2007 13:25:48 -0600
Subject: [libxml-devel] Use xfree not free
Message-ID: <46928BBC.5070502@savagexi.com>

A couple of weeks ago Johnathan Conley and I spent some time trying to 
get libxml to work better on windows.  Unfortunately, we did not succeed 
due to the memory leaks mentioned in other threads.

However, one obvious error in the current bindings is the use of free. 
The bindings use Ruby's ALLOCA macro to allocate memory for various 
objects, but then proceed to unallocate using free instead of xfree 
(xfree is also provided by Ruby).

The technical reason for this is that ALLOCA and xfree create and free 
memory from the Ruby executable's stack.  On the other hand, malloc and 
free in an extension create and free memory from the extension's stack. 
  In some cases the executable's and extension's stack are one and the 
same thing.

But on windows they are often not.  In this case, Ruby is built with 
VC++ 6.0 but I build the libxml bindings with VC++2005.  That means that 
the Ruby executable and extension library use different versions of the 
C runtime library, and thus have different stacks.  For more information 
see http://msdn2.microsoft.com/en-us/library/ms235460(VS.80).aspx.

The end result is that ALLOCA uses memory from the executable's stack, 
and then free unallocates them using the extension's stack.  And of 
course you get segmentation faults.

The solution to this one is easy - all free's should be changed to xfree's.

Thanks,

Charlie

P.S.  - Any progress on integrating the leak patches from OpenStreetMap?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3237 bytes
Desc: S/MIME Cryptographic Signature
Url : http://rubyforge.org/pipermail/libxml-devel/attachments/20070709/eeacf7ee/attachment.bin 

From transfire at gmail.com  Wed Jul 11 15:53:10 2007
From: transfire at gmail.com (TRANS)
Date: Wed, 11 Jul 2007 15:53:10 -0400
Subject: [libxml-devel] Use xfree not free
In-Reply-To: <46928BBC.5070502@savagexi.com>
References: <46928BBC.5070502@savagexi.com>
Message-ID: <4b6f054f0707111253l20242632s19582bde19328261@mail.gmail.com>

On 7/9/07, Charlie Savage <cfis at savagexi.com> wrote:

> P.S.  - Any progress on integrating the leak patches from OpenStreetMap?

Hi--

I haven't heard a peep from Ross in a long time. As the project's,
sort of, "guardian angel" I think its time for a new maintainer*. Any
takers?

T.

*I'm not a C coder, which is why I can't do it.

From cfis at savagexi.com  Wed Jul 11 15:55:40 2007
From: cfis at savagexi.com (Charlie Savage)
Date: Wed, 11 Jul 2007 13:55:40 -0600
Subject: [libxml-devel] Use xfree not free
In-Reply-To: <4b6f054f0707111253l20242632s19582bde19328261@mail.gmail.com>
References: <46928BBC.5070502@savagexi.com>
	<4b6f054f0707111253l20242632s19582bde19328261@mail.gmail.com>
Message-ID: <469535BC.9030602@savagexi.com>

Hi Trans,

If I had time, I'd be happy to, but unfortunately I don't really.

But if you add me on as a committer, I'll be happy to make the 
free/xfree changes.

And I'm happy to apply patches if people can provide test cases showing 
what's broken and that their patch fixes it.  It seems like the biggest 
issue is the memory leaks...any progress on those?

Charlie


TRANS wrote:
> On 7/9/07, Charlie Savage <cfis at savagexi.com> wrote:
> 
>> P.S.  - Any progress on integrating the leak patches from OpenStreetMap?
> 
> Hi--
> 
> I haven't heard a peep from Ross in a long time. As the project's,
> sort of, "guardian angel" I think its time for a new maintainer*. Any
> takers?
> 
> T.
> 
> *I'm not a C coder, which is why I can't do it.
> _______________________________________________
> libxml-devel mailing list
> libxml-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/libxml-devel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3237 bytes
Desc: S/MIME Cryptographic Signature
Url : http://rubyforge.org/pipermail/libxml-devel/attachments/20070711/d149efb5/attachment.bin 

From cfis at savagexi.com  Wed Jul 11 15:58:17 2007
From: cfis at savagexi.com (Charlie Savage)
Date: Wed, 11 Jul 2007 13:58:17 -0600
Subject: [libxml-devel] Use xfree not free
In-Reply-To: <4b6f054f0707111253l20242632s19582bde19328261@mail.gmail.com>
References: <46928BBC.5070502@savagexi.com>
	<4b6f054f0707111253l20242632s19582bde19328261@mail.gmail.com>
Message-ID: <46953659.6090403@savagexi.com>

And I'll happy to make it build on Windows - I've mastered that skill 
with ruby-prof :)

Charlie

TRANS wrote:
> On 7/9/07, Charlie Savage <cfis at savagexi.com> wrote:
> 
>> P.S.  - Any progress on integrating the leak patches from OpenStreetMap?
> 
> Hi--
> 
> I haven't heard a peep from Ross in a long time. As the project's,
> sort of, "guardian angel" I think its time for a new maintainer*. Any
> takers?
> 
> T.
> 
> *I'm not a C coder, which is why I can't do it.
> _______________________________________________
> libxml-devel mailing list
> libxml-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/libxml-devel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3237 bytes
Desc: S/MIME Cryptographic Signature
Url : http://rubyforge.org/pipermail/libxml-devel/attachments/20070711/305e8241/attachment.bin 

From transfire at gmail.com  Wed Jul 11 16:01:00 2007
From: transfire at gmail.com (TRANS)
Date: Wed, 11 Jul 2007 16:01:00 -0400
Subject: [libxml-devel] Use xfree not free
In-Reply-To: <469535BC.9030602@savagexi.com>
References: <46928BBC.5070502@savagexi.com>
	<4b6f054f0707111253l20242632s19582bde19328261@mail.gmail.com>
	<469535BC.9030602@savagexi.com>
Message-ID: <4b6f054f0707111301x61b0b6dard9a79e2270957b40@mail.gmail.com>

On 7/11/07, Charlie Savage <cfis at savagexi.com> wrote:
> Hi Trans,
>
> If I had time, I'd be happy to, but unfortunately I don't really.
>
> But if you add me on as a committer, I'll be happy to make the
> free/xfree changes.

Cool. What's your rubyforge id?

T.

From cfis at savagexi.com  Wed Jul 11 16:04:05 2007
From: cfis at savagexi.com (Charlie Savage)
Date: Wed, 11 Jul 2007 14:04:05 -0600
Subject: [libxml-devel] Use xfree not free
In-Reply-To: <4b6f054f0707111301x61b0b6dard9a79e2270957b40@mail.gmail.com>
References: <46928BBC.5070502@savagexi.com>	<4b6f054f0707111253l20242632s19582bde19328261@mail.gmail.com>	<469535BC.9030602@savagexi.com>
	<4b6f054f0707111301x61b0b6dard9a79e2270957b40@mail.gmail.com>
Message-ID: <469537B5.3000306@savagexi.com>

> Cool. What's your rubyforge id?

cfis - http://rubyforge.org/users/cfis/

Charlie
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3237 bytes
Desc: S/MIME Cryptographic Signature
Url : http://rubyforge.org/pipermail/libxml-devel/attachments/20070711/e7c08d30/attachment.bin 

From deliverable at gmail.com  Thu Jul 12 09:57:40 2007
From: deliverable at gmail.com (Alexy Khrabrov)
Date: Thu, 12 Jul 2007 17:57:40 +0400
Subject: [libxml-devel] object_id's differ for the same (root) node in
	ancestor lists
In-Reply-To: <469537B5.3000306@savagexi.com>
References: <46928BBC.5070502@savagexi.com>	<4b6f054f0707111253l20242632s19582bde19328261@mail.gmail.com>	<469535BC.9030602@savagexi.com>
	<4b6f054f0707111301x61b0b6dard9a79e2270957b40@mail.gmail.com>
	<469537B5.3000306@savagexi.com>
Message-ID: <B14052C2-43F8-4E36-877D-7CE980C331DA@gmail.com>

I've written some simple routines to walk an XML ontology with xml/ 
libxml, and noticed that putting XML::Node objects into lists and  
returning them may lead to a situation where two lists contain the  
same node with different object_id's.  Here's an exmple, funstion  
ancestors:

require 'xml/libxml'

# extend XML::Node with helper methods
class XML::Node
   def ancestors(node_type=nil)
     #print "=>"
     res = []
     p = self
     while p = p.parent and p.name do
       next if node_type and p.name != node_type
       res << p if p
       #puts "added #{p.name}"
     end
     res.reverse
   end

   # ... -- more methods
end

Now I found that if I call it on some tree and get at least two  
nodes, and then compare the first elements of each list, which should  
be the root in both cases, I get different object_id's.  But ==  
returns equality.  What's going on here?  Hpricot used to maintain  
the same object_id across assignments, which is usually the case in  
Ruby...  Are there any think objects created for XML::Node's here?

Cheers,
Alexy 

From cfis at savagexi.com  Thu Jul 12 12:18:30 2007
From: cfis at savagexi.com (Charlie Savage)
Date: Thu, 12 Jul 2007 10:18:30 -0600
Subject: [libxml-devel] object_id's differ for the same (root) node in
 ancestor lists
In-Reply-To: <B14052C2-43F8-4E36-877D-7CE980C331DA@gmail.com>
References: <46928BBC.5070502@savagexi.com>	<4b6f054f0707111253l20242632s19582bde19328261@mail.gmail.com>	<469535BC.9030602@savagexi.com>	<4b6f054f0707111301x61b0b6dard9a79e2270957b40@mail.gmail.com>	<469537B5.3000306@savagexi.com>
	<B14052C2-43F8-4E36-877D-7CE980C331DA@gmail.com>
Message-ID: <46965456.8020504@savagexi.com>

Hi Alexy,

I'm not familiar with the libxml binding code, but have worked a fair 
bit on the SWIG ruby bindings in the past.

Unless you try hard not to - you'll always end with multiple Ruby 
objects referencing the same C object.  Just think of a C object that 
has some property.  Say a node has attributes.  Every time you call:

node->attributes

You'll create a new wrapper object.  To avoid that, you need to cache 
the previous wrapper object you created and reuse it. That  makes the 
code uglier and more complex.  But the payoff is that it seems more 
natural from the Ruby side, and it makes memory management a whole lot 
easier.

 From reading previous posts in this mailing list, it seems like libxml 
probably hast this issue in spades.

Would be interesting to see how Hpricot avoids it.

Charlie

Alexy Khrabrov wrote:
> I've written some simple routines to walk an XML ontology with xml/ 
> libxml, and noticed that putting XML::Node objects into lists and  
> returning them may lead to a situation where two lists contain the  
> same node with different object_id's.  Here's an exmple, funstion  
> ancestors:
> 
> require 'xml/libxml'
> 
> # extend XML::Node with helper methods
> class XML::Node
>    def ancestors(node_type=nil)
>      #print "=>"
>      res = []
>      p = self
>      while p = p.parent and p.name do
>        next if node_type and p.name != node_type
>        res << p if p
>        #puts "added #{p.name}"
>      end
>      res.reverse
>    end
> 
>    # ... -- more methods
> end
> 
> Now I found that if I call it on some tree and get at least two  
> nodes, and then compare the first elements of each list, which should  
> be the root in both cases, I get different object_id's.  But ==  
> returns equality.  What's going on here?  Hpricot used to maintain  
> the same object_id across assignments, which is usually the case in  
> Ruby...  Are there any think objects created for XML::Node's here?
> 
> Cheers,
> Alexy 
> _______________________________________________
> libxml-devel mailing list
> libxml-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/libxml-devel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3237 bytes
Desc: S/MIME Cryptographic Signature
Url : http://rubyforge.org/pipermail/libxml-devel/attachments/20070712/a65d640a/attachment.bin 

From transfire at gmail.com  Thu Jul 12 13:03:21 2007
From: transfire at gmail.com (TRANS)
Date: Thu, 12 Jul 2007 13:03:21 -0400
Subject: [libxml-devel] Maintainer
Message-ID: <4b6f054f0707121003v210d7b10sa49519717c70f73f@mail.gmail.com>

I see that Ross Bramford is not even listed as a project member any
more --let alone the maintainer. So that means things fall back to
Sean Chittenden -- but is he around? I see three other members listed:

  Dan Janowski
  Laurent Sansonetti
  Pat Eyler

Are any of you guys around?

I fear there's no one at the hem again, but there are still important
issues that need to be addressed. From the looks of it there hasn't
been a commit since January, though there have been patches offered up
on this list since then.

What's the word?

T.

From matthew at ueckerman.net  Sun Jul 15 23:24:58 2007
From: matthew at ueckerman.net (matthew at ueckerman.net)
Date: Sun, 15 Jul 2007 23:24:58 -0400
Subject: [libxml-devel] Seg fault during parse on Solaris.
Message-ID: <380-22007711632458443@M2W044.mail2web.com>

With libxml-ruby 0.3.8.4 and SunOS 5.10 when parsing multiple strings or
files concurrently I occasionally observe the following seg fault:

Program received signal SIGSEGV, Segmentation fault.
0xfe813874 in xmlParseDocument () from /usr/lib/libxml2.so.2
(gdb) where
#0  0xfe813874 in xmlParseDocument () from /usr/lib/libxml2.so.2
#1  0xfe8f2bc5 in ruby_xml_parser_parse (self=139580352) at
ruby_xml_parser.c:1125
#2  0x08063c38 in rb_call0 (klass=159390400, recv=162825420, id=13289,
oid=4270630784, argc=0, argv=0x0, body=0x9801160, 
    flags=0) at eval.c:5665
#3  0x08063f87 in rb_call (klass=159390400, recv=162825420, mid=13289,
argc=0, argv=0x0, scope=0) at eval.c:6048
#4  0x080600e1 in rb_eval (self=161830460, n=0xfe8ca780) at ruby.h:654

Here's a derivative of the source:

overall_content = []
document = XML::Parser.file(<some file path>).parse
document.find(<some xpath>).each do |outer_element|
  outer_element.find(<some xpath>).each do |inner_element|
    if inner_element.property(<some attribute>)
      overall_content << inner_element.content.strip
    end
  end
end

Anyone with any idea of the cause?  It doesn't appear to be a known issue
based on the open defects and list archive.

Kind Regards,

Matthew Ueckerman

--------------------------------------------------------------------
mail2web.com ? What can On Demand Business Solutions do for you?
http://link.mail2web.com/Business/SharePoint



From matthew at ueckerman.net  Mon Jul 16 09:12:32 2007
From: matthew at ueckerman.net (Matthew Ueckerman)
Date: Mon, 16 Jul 2007 23:12:32 +1000
Subject: [libxml-devel] Seg fault during parse on Solaris.
References: <380-22007711632458443@M2W044.mail2web.com>
Message-ID: <000001c7c872$b8500f70$4201a8c0@DUECKES>

Actually I stand corrected, a patch was submitted to this list some months
ago that appears to be related:

http://rubyforge.org/pipermail/libxml-devel/2007-March/000288.html

Has this patch had any traction?

Kind Regards,

Matthew Ueckerman

----- Original Message ----- 
From: <matthew at ueckerman.net>
To: <libxml-devel at rubyforge.org>
Sent: Monday, July 16, 2007 1:24 PM
Subject: [libxml-devel] Seg fault during parse on Solaris.


With libxml-ruby 0.3.8.4 and SunOS 5.10 when parsing multiple strings or
files concurrently I occasionally observe the following seg fault:

Program received signal SIGSEGV, Segmentation fault.
0xfe813874 in xmlParseDocument () from /usr/lib/libxml2.so.2
(gdb) where
#0  0xfe813874 in xmlParseDocument () from /usr/lib/libxml2.so.2
#1  0xfe8f2bc5 in ruby_xml_parser_parse (self=139580352) at
ruby_xml_parser.c:1125
#2  0x08063c38 in rb_call0 (klass=159390400, recv=162825420, id=13289,
oid=4270630784, argc=0, argv=0x0, body=0x9801160,
    flags=0) at eval.c:5665
#3  0x08063f87 in rb_call (klass=159390400, recv=162825420, mid=13289,
argc=0, argv=0x0, scope=0) at eval.c:6048
#4  0x080600e1 in rb_eval (self=161830460, n=0xfe8ca780) at ruby.h:654

Here's a derivative of the source:

overall_content = []
document = XML::Parser.file(<some file path>).parse
document.find(<some xpath>).each do |outer_element|
  outer_element.find(<some xpath>).each do |inner_element|
    if inner_element.property(<some attribute>)
      overall_content << inner_element.content.strip
    end
  end
end

Anyone with any idea of the cause?  It doesn't appear to be a known issue
based on the open defects and list archive.

Kind Regards,

Matthew Ueckerman

--------------------------------------------------------------------
mail2web.com - What can On Demand Business Solutions do for you?
http://link.mail2web.com/Business/SharePoint


_______________________________________________
libxml-devel mailing list
libxml-devel at rubyforge.org
http://rubyforge.org/mailman/listinfo/libxml-devel

From transfire at gmail.com  Wed Jul 18 08:53:46 2007
From: transfire at gmail.com (TRANS)
Date: Wed, 18 Jul 2007 08:53:46 -0400
Subject: [libxml-devel] Maintainer
In-Reply-To: <4b6f054f0707121003v210d7b10sa49519717c70f73f@mail.gmail.com>
References: <4b6f054f0707121003v210d7b10sa49519717c70f73f@mail.gmail.com>
Message-ID: <4b6f054f0707180553r5cf3534fx59546b77579179a6@mail.gmail.com>

On 7/12/07, TRANS <transfire at gmail.com> wrote:
> I see that Ross Bramford is not even listed as a project member any
> more --let alone the maintainer. So that means things fall back to
> Sean Chittenden -- but is he around? I see three other members listed:
>
>  Dan Janowski
>  Laurent Sansonetti
>  Pat Eyler
>
> Are any of you guys around?
>
> I fear there's no one at the hem again, but there are still important
> issues that need to be addressed. From the looks of it there hasn't
> been a commit since January, though there have been patches offered up
> on this list since then.
>
> What's the word?

Anyone? Is my message getting through?

T.

From has.sox at gmail.com  Wed Jul 18 09:04:00 2007
From: has.sox at gmail.com (Daniel N)
Date: Wed, 18 Jul 2007 23:04:00 +1000
Subject: [libxml-devel] Maintainer
In-Reply-To: <4b6f054f0707180553r5cf3534fx59546b77579179a6@mail.gmail.com>
References: <4b6f054f0707121003v210d7b10sa49519717c70f73f@mail.gmail.com>
	<4b6f054f0707180553r5cf3534fx59546b77579179a6@mail.gmail.com>
Message-ID: <2fff50390707180604y74b5b5c2g9e69e42b4f26b4c8@mail.gmail.com>

On 7/18/07, TRANS <transfire at gmail.com> wrote:
>
> On 7/12/07, TRANS <transfire at gmail.com> wrote:
> > I see that Ross Bramford is not even listed as a project member any
> > more --let alone the maintainer. So that means things fall back to
> > Sean Chittenden -- but is he around? I see three other members listed:
> >
> >  Dan Janowski
> >  Laurent Sansonetti
> >  Pat Eyler
> >
> > Are any of you guys around?
> >
> > I fear there's no one at the hem again, but there are still important
> > issues that need to be addressed. From the looks of it there hasn't
> > been a commit since January, though there have been patches offered up
> > on this list since then.
> >
> > What's the word?
>
> Anyone? Is my message getting through?
>
> T.



I have not had any response from a maintainer of this project either.

Your message is getting through though.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/libxml-devel/attachments/20070718/4bdebb98/attachment.html 

From keith at oreilly.com  Wed Jul 18 09:03:01 2007
From: keith at oreilly.com (Keith Fahlgren)
Date: Wed, 18 Jul 2007 09:03:01 -0400
Subject: [libxml-devel] Maintainer
In-Reply-To: <4b6f054f0707180553r5cf3534fx59546b77579179a6@mail.gmail.com>
References: <4b6f054f0707121003v210d7b10sa49519717c70f73f@mail.gmail.com>
	<4b6f054f0707180553r5cf3534fx59546b77579179a6@mail.gmail.com>
Message-ID: <469E0F85.8090504@oreilly.com>

On 7/18/07 8:53 AM, TRANS wrote:
> On 7/12/07, TRANS <transfire at gmail.com> wrote:
>> I see that Ross Bramford is not even listed as a project member any
>> more --let alone the maintainer. So that means things fall back to
>> Sean Chittenden -- but is he around? I see three other members listed:
>>
>>  Dan Janowski
>>  Laurent Sansonetti
>>  Pat Eyler
>>
>> Are any of you guys around?
>>
>> I fear there's no one at the hem again, but there are still important
>> issues that need to be addressed. From the looks of it there hasn't
>> been a commit since January, though there have been patches offered up
>> on this list since then.
>>
>> What's the word?
> 
> Anyone? Is my message getting through?

Yes.

From lrz at chopine.be  Wed Jul 18 09:34:55 2007
From: lrz at chopine.be (Laurent Sansonetti)
Date: Wed, 18 Jul 2007 15:34:55 +0200
Subject: [libxml-devel] Maintainer
In-Reply-To: <4b6f054f0707180553r5cf3534fx59546b77579179a6@mail.gmail.com>
References: <4b6f054f0707121003v210d7b10sa49519717c70f73f@mail.gmail.com>
	<4b6f054f0707180553r5cf3534fx59546b77579179a6@mail.gmail.com>
Message-ID: <A1EA6DE6-650B-465C-94C0-0B1C9215F189@chopine.be>


On Jul 18, 2007, at 2:53 PM, TRANS wrote:

> On 7/12/07, TRANS <transfire at gmail.com> wrote:
>> I see that Ross Bramford is not even listed as a project member any
>> more --let alone the maintainer. So that means things fall back to
>> Sean Chittenden -- but is he around? I see three other members  
>> listed:
>>
>> Dan Janowski
>> Laurent Sansonetti
>> Pat Eyler
>>
>> Are any of you guys around?
>>
>> I fear there's no one at the hem again, but there are still important
>> issues that need to be addressed. From the looks of it there hasn't
>> been a commit since January, though there have been patches offered  
>> up
>> on this list since then.
>>
>> What's the word?
>
> Anyone? Is my message getting through?

Personally, I unfortunately do not have time to spend on this project.  
I can just promise to help maintaining the XML::Reader API.

I nevertheless hope that someone will take the maintenance back,  
because libxml-ruby is a crucial project.

Laurent

From bascule at gmail.com  Wed Jul 18 21:27:42 2007
From: bascule at gmail.com (Tony)
Date: Wed, 18 Jul 2007 19:27:42 -0600
Subject: [libxml-devel] Builder-compatible API
Message-ID: <c7e6b2b00707181827g24d77b8fi8ca4decc3843c657@mail.gmail.com>

Not sure how much interest there is in this, but after being frustrated with
the performance of Builder I whipped up a mostly Builder-compatible frontend
to libxml-ruby.  My initial benchmarks show it to be about twice as fast as
builder on large documents:

http://pastie.caboo.se/79575

libxml-ruby based proof-of-concept:
>> puts Benchmark.measure { xml = XmlBase.new; xml.foo(:a => :b) { 100.times{
xml.bar(:c => :d) { 1000.times { xml.baz("Text goes here") } } } }.to_s }
  3.130000   1.290000   4.420000 (  4.433537)
  2.850000   1.280000   4.130000 (  4.133377)
  2.810000   1.250000   4.060000 (  4.070264)

Builder:
>> puts Benchmark.measure { xml = Builder::XmlMarkup.new ; xml.foo(:a => :b)
{ 100.times { xml.bar(:c => :d) { 1000.times { xml.baz("Text goes here") } }
} } }
  9.300000   0.030000   9.330000 (  9.374801)
  9.290000   0.020000   9.310000 (  9.315121)
  9.280000   0.020000   9.300000 (  9.316049)

Tony Arcieri
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/libxml-devel/attachments/20070718/ca511923/attachment.html 

From rikkus at gmail.com  Thu Jul 19 07:15:03 2007
From: rikkus at gmail.com (Rik Hemsley)
Date: Thu, 19 Jul 2007 12:15:03 +0100
Subject: [libxml-devel] Builder-compatible API
In-Reply-To: <c7e6b2b00707181827g24d77b8fi8ca4decc3843c657@mail.gmail.com>
References: <c7e6b2b00707181827g24d77b8fi8ca4decc3843c657@mail.gmail.com>
Message-ID: <868dee660707190415u40a85398i4b518b95c7baa98c@mail.gmail.com>

On 19/07/07, Tony <bascule at gmail.com> wrote:
>
> Not sure how much interest there is in this, but after being frustrated
> with the performance of Builder I whipped up a mostly Builder-compatible
> frontend to libxml-ruby.  My initial benchmarks show it to be about twice as
> fast as builder on large documents:
>
> http://pastie.caboo.se/79575


Nice. I noticed Build was slow, too. Looking forward to seeing it!

Rik
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/libxml-devel/attachments/20070719/caf59244/attachment.html 

From tony at clickcaster.com  Thu Jul 19 13:15:36 2007
From: tony at clickcaster.com (Tony Arcieri)
Date: Thu, 19 Jul 2007 11:15:36 -0600
Subject: [libxml-devel] Builder-compatible API
In-Reply-To: <868dee660707190415u40a85398i4b518b95c7baa98c@mail.gmail.com>
References: <c7e6b2b00707181827g24d77b8fi8ca4decc3843c657@mail.gmail.com>
	<868dee660707190415u40a85398i4b518b95c7baa98c@mail.gmail.com>
Message-ID: <c7e6b2b00707191015x7f2a144dk2b728e8bb5b2b1b9@mail.gmail.com>

Well, I was excited about this idea, but started getting diminishing
returns, at least in the test cases I was using:

The basic pattern is:

xml = Builder::XmlMarkup.new ; xml.foo(:a => :b) { N.times { xml.bar(:c =>
:d) { 1000.times { xml.baz("Text goes here") } } } }

with N increasing.  This also includes a .to_s on the libxml Builder version
which outputs the actual markup.

200 children w\ 1000 grandchildren each:

Builder: 18.560000   0.030000  18.590000 ( 18.593951)
libxml:    9.030000   5.410000  14.440000 ( 14.485134)

Where libxml Builder was twice as fast on the original benchmarks I posted,
here it's only marginally faster.

300 children w\ 1000 grandchildren each:

Builder: 27.860000   0.040000  27.900000 ( 27.905695)
libxml:  17.760000  12.160000  29.920000 ( 29.931187)

Here Builder has surpassed libxml Builder.

400 children w\ 1000 grandchildren each:

  37.150000   0.050000  37.200000 ( 37.204134)
 28.380000  21.610000  49.990000 ( 50.522918)

I didn't go beyond that.

Anyway, any ideas what the slowdown might be?  Seems like the native
extension bit isn't scaling linearly with the document size.

On 7/19/07, Rik Hemsley <rikkus at gmail.com> wrote:
>
> On 19/07/07, Tony <bascule at gmail.com> wrote:
> >
> > Not sure how much interest there is in this, but after being frustrated
> > with the performance of Builder I whipped up a mostly Builder-compatible
> > frontend to libxml-ruby.  My initial benchmarks show it to be about twice as
> > fast as builder on large documents:
> >
> > http://pastie.caboo.se/79575
>
>
> Nice. I noticed Build was slow, too. Looking forward to seeing it!
>
> Rik
>
>
>
>
> _______________________________________________
> libxml-devel mailing list
> libxml-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/libxml-devel
>



-- 
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
720-227-0129 ext. 202
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/libxml-devel/attachments/20070719/8e6950dc/attachment.html 

From transfire at gmail.com  Thu Jul 19 18:40:38 2007
From: transfire at gmail.com (TRANS)
Date: Thu, 19 Jul 2007 15:40:38 -0700
Subject: [libxml-devel] Builder-compatible API
In-Reply-To: <c7e6b2b00707191015x7f2a144dk2b728e8bb5b2b1b9@mail.gmail.com>
References: <c7e6b2b00707181827g24d77b8fi8ca4decc3843c657@mail.gmail.com>
	<868dee660707190415u40a85398i4b518b95c7baa98c@mail.gmail.com>
	<c7e6b2b00707191015x7f2a144dk2b728e8bb5b2b1b9@mail.gmail.com>
Message-ID: <4b6f054f0707191540l51c034f0w33889f328358d063@mail.gmail.com>

On 7/19/07, Tony Arcieri <tony at clickcaster.com> wrote:
> Well, I was excited about this idea, but started getting diminishing
> returns, at least in the test cases I was using:
>
> The basic pattern is:
>
> xml = Builder::XmlMarkup.new ; xml.foo(:a => :b) { N.times { xml.bar (:c =>
> :d) { 1000.times { xml.baz("Text goes here") } } } }
>
> with N increasing.  This also includes a .to_s on the libxml Builder version
> which outputs the actual markup.
>
> 200 children w\ 1000 grandchildren each:
>
> Builder: 18.560000   0.030000  18.590000 ( 18.593951)
> libxml:    9.030000   5.410000  14.440000 ( 14.485134)
>
> Where libxml Builder was twice as fast on the original benchmarks I posted,
> here it's only marginally faster.
>
> 300 children w\ 1000 grandchildren each:
>
> Builder: 27.860000   0.040000  27.900000 ( 27.905695)
> libxml:  17.760000  12.160000  29.920000 ( 29.931187)
>
> Here Builder has surpassed libxml Builder.
>
>  400 children w\ 1000 grandchildren each:
>
>   37.150000   0.050000  37.200000 ( 37.204134)
>  28.380000  21.610000  49.990000 ( 50.522918)
>
> I didn't go beyond that.
>
> Anyway, any ideas what the slowdown might be?  Seems like the native
> extension bit isn't scaling linearly with the document size.

I'm not sure, but I created a template system that could used either
libxml or rexml on the back-end. Disturbingly rexml was faster!
Something is very wrong.

T.

From rosco at roscopeco.co.uk  Sat Jul 21 04:45:42 2007
From: rosco at roscopeco.co.uk (Ross Bamford)
Date: Sat, 21 Jul 2007 09:45:42 +0100
Subject: [libxml-devel] Maintainer
In-Reply-To: <4b6f054f0707121003v210d7b10sa49519717c70f73f@mail.gmail.com>
References: <4b6f054f0707121003v210d7b10sa49519717c70f73f@mail.gmail.com>
Message-ID: <op.tvs3ygy8kc9f7k@localhost.localdomain>

Hi Folks,

On Thu, 12 Jul 2007 18:03:21 +0100, TRANS <transfire at gmail.com> wrote:

> I see that Ross Bramford is not even listed as a project member any
> more --let alone the maintainer. So that means things fall back to
> Sean Chittenden -- but is he around? I see three other members listed:
>
>   Dan Janowski
>   Laurent Sansonetti
>   Pat Eyler
>
> Are any of you guys around?
>
> I fear there's no one at the hem again, but there are still important
> issues that need to be addressed. From the looks of it there hasn't
> been a commit since January, though there have been patches offered up
> on this list since then.
>
> What's the word?
>

Well, I was removed from the project a while ago, I think when it became  
clear that circumstances wouldn't allow me to put in the time even to  
apply patches anymore - I've been unbelivably busy these past months with  
work, and although I hoped to still have a little time to spend on my OSS  
commitments, my now-five-month-old daughter had other plans :)

Things are a little quieter now (on the work front at least) and I am  
planning to get back into things, but I do honestly doubt I'd be able to  
devote as much time as previously. I haven't heard from Sean for a while  
(my last few emails to him went unanswered) and there are still a number  
of fundamental problems with libxml-ruby - mostly to do with the fact that  
libxml and ruby are constantly fighting over memory management and the  
like.

I want to say I'd come back to the project, but I simply don't have the  
time to do it justice as sole maintainer. When I first joined libxml-ruby  
I said that I didn't want to lead the project, but was happy to help. Over  
time, as people's commitments changed, that's pretty much what happened.  
The result is that when my circumstances changed suddenly (as they often  
do) the project was left in the crisis we have today.

I would be willing to come back in a 'code-caretaker' capacity -  
committing patches and the like, perhaps fixing the odd bug, but that  
still leaves a lot of work that needs to be done: someone needs to watch  
the trackers, and maybe filter out 'not a bug's or do preliminary  
investigation on new bugs/feature reqs. The forums on Rubyforge seem to be  
filling up somewhat. Test coverage needs to be improved. The project needs  
to be promoted, and whatever passes for 'PR' in open-source needs to be  
taken care of in general. These are all things I can't guarantee any time  
for, and things that don't involve writing C.

The Windows issue matters, too, but since I've been lucky enough to avoid  
Windows almost entirely in my professional life for several years now I  
don't really know the platform well enough to handle this, and frankly I'm  
unwilling to take the time to learn for the sake of compatibility on one  
project. Another thing that needs to be done.

A lot of good work was done on libxml-ruby, as recently as Laurent's  
reader contributions, and it'd be a shame to see things grind to a halt.  
I'm willing to help make sure they don't, but what we really need is  
people involved in all that other stuff I talked about above.

-- 
Ross Bamford - rosco at roscopeco.co.uk

From cfis at savagexi.com  Sun Jul 22 17:02:43 2007
From: cfis at savagexi.com (Charlie Savage)
Date: Sun, 22 Jul 2007 15:02:43 -0600
Subject: [libxml-devel] Use xfree not free
In-Reply-To: <4b6f054f0707111301x61b0b6dard9a79e2270957b40@mail.gmail.com>
References: <46928BBC.5070502@savagexi.com>	<4b6f054f0707111253l20242632s19582bde19328261@mail.gmail.com>	<469535BC.9030602@savagexi.com>
	<4b6f054f0707111301x61b0b6dard9a79e2270957b40@mail.gmail.com>
Message-ID: <46A3C5F3.6080309@savagexi.com>

Hi TRANS,

Were you still planning on doing this?  If so, I'll patch ruby-prof to 
work better on Windows.  And I can package up a windows gem.


Charlie

TRANS wrote:
> On 7/11/07, Charlie Savage <cfis at savagexi.com> wrote:
>> Hi Trans,
>>
>> If I had time, I'd be happy to, but unfortunately I don't really.
>>
>> But if you add me on as a committer, I'll be happy to make the
>> free/xfree changes.
> 
> Cool. What's your rubyforge id?
> 
> T.
> _______________________________________________
> libxml-devel mailing list
> libxml-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/libxml-devel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3237 bytes
Desc: S/MIME Cryptographic Signature
Url : http://rubyforge.org/pipermail/libxml-devel/attachments/20070722/818e1c13/attachment.bin 

From cfis at savagexi.com  Sun Jul 22 17:06:32 2007
From: cfis at savagexi.com (Charlie Savage)
Date: Sun, 22 Jul 2007 15:06:32 -0600
Subject: [libxml-devel] Maintainer
In-Reply-To: <op.tvs3ygy8kc9f7k@localhost.localdomain>
References: <4b6f054f0707121003v210d7b10sa49519717c70f73f@mail.gmail.com>
	<op.tvs3ygy8kc9f7k@localhost.localdomain>
Message-ID: <46A3C6D8.6010909@savagexi.com>

> there are still a number  
> of fundamental problems with libxml-ruby - mostly to do with the fact that  
> libxml and ruby are constantly fighting over memory management and the  
> like.

Ross - Could you summarize these issues, and exactly what is going wrong 
  where?  It would be extremely helpful as a guide to anyone who has 
time to dig into the issue.

I did a lot of work on SWIG's ruby support, so have a good feeling for 
the various memory issues that can arise.  But I know nothing about 
libxml's api or memory management, or how the ruby bindings interact 
with it.

Also, does the patch from open street map fix the issue?

> The Windows issue matters, too, but since I've been lucky enough to avoid  
> Windows almost entirely in my professional life for several years now I  
> don't really know the platform well enough to handle this, and frankly I'm  
> unwilling to take the time to learn for the sake of compatibility on one  
> project. Another thing that needs to be done.

I can solve the windows issues.  I have a patch to fix up memory 
allocation on windows, and with my experience with ruby-prof, I can 
easily create a win32 gem.

> A lot of good work was done on libxml-ruby, as recently as Laurent's  
> reader contributions, and it'd be a shame to see things grind to a halt.  
> I'm willing to help make sure they don't, but what we really need is  
> people involved in all that other stuff I talked about above.

Thanks,

Charlie
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3237 bytes
Desc: S/MIME Cryptographic Signature
Url : http://rubyforge.org/pipermail/libxml-devel/attachments/20070722/f0303d16/attachment.bin 

From cfis at savagexi.com  Sun Jul 22 17:07:09 2007
From: cfis at savagexi.com (Charlie Savage)
Date: Sun, 22 Jul 2007 15:07:09 -0600
Subject: [libxml-devel] Use xfree not free
In-Reply-To: <46A3C5F3.6080309@savagexi.com>
References: <46928BBC.5070502@savagexi.com>	<4b6f054f0707111253l20242632s19582bde19328261@mail.gmail.com>	<469535BC.9030602@savagexi.com>	<4b6f054f0707111301x61b0b6dard9a79e2270957b40@mail.gmail.com>
	<46A3C5F3.6080309@savagexi.com>
Message-ID: <46A3C6FD.8000000@savagexi.com>

> Hi TRANS,
> 
> Were you still planning on doing this?  If so, I'll patch ruby-prof to 
> work better on Windows.  And I can package up a windows gem.

Right - that was libxml not ruby-prof...

Charlie
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3237 bytes
Desc: S/MIME Cryptographic Signature
Url : http://rubyforge.org/pipermail/libxml-devel/attachments/20070722/6722f136/attachment.bin 

From transfire at gmail.com  Sun Jul 22 20:18:17 2007
From: transfire at gmail.com (TRANS)
Date: Sun, 22 Jul 2007 17:18:17 -0700
Subject: [libxml-devel] Use xfree not free
In-Reply-To: <46A3C5F3.6080309@savagexi.com>
References: <46928BBC.5070502@savagexi.com>
	<4b6f054f0707111253l20242632s19582bde19328261@mail.gmail.com>
	<469535BC.9030602@savagexi.com>
	<4b6f054f0707111301x61b0b6dard9a79e2270957b40@mail.gmail.com>
	<46A3C5F3.6080309@savagexi.com>
Message-ID: <4b6f054f0707221718i233bf7b0va590acaffd6f9ca9@mail.gmail.com>

On 7/22/07, Charlie Savage <cfis at savagexi.com> wrote:
> Hi TRANS,
>
> Were you still planning on doing this?  If so, I'll patch ruby-prof to
> work better on Windows.  And I can package up a windows gem.

Yes, but right now I have to work through Red tape. Seems Ross removed
himself from teh members list, and the only person with admin
privliges is Sean, who once again can't be found. So I'm talking to
Tom Copeland to see if I can get added as admin.

Might take a few more days to get straight. I'll keep you posted.

T.

From rosco at roscopeco.co.uk  Mon Jul 23 06:31:02 2007
From: rosco at roscopeco.co.uk (Ross Bamford)
Date: Mon, 23 Jul 2007 11:31:02 +0100
Subject: [libxml-devel] Maintainer
In-Reply-To: <46A3C6D8.6010909@savagexi.com>
References: <4b6f054f0707121003v210d7b10sa49519717c70f73f@mail.gmail.com>
	<op.tvs3ygy8kc9f7k@localhost.localdomain>
	<46A3C6D8.6010909@savagexi.com>
Message-ID: <op.tvwx50ivkc9f7k@localhost.localdomain>

On Sun, 22 Jul 2007 22:06:32 +0100, Charlie Savage <cfis at savagexi.com>  
wrote:

>> there are still a number
>> of fundamental problems with libxml-ruby - mostly to do with the fact  
>> that
>> libxml and ruby are constantly fighting over memory management and the
>> like.
>
> Ross - Could you summarize these issues, and exactly what is going wrong
>   where?  It would be extremely helpful as a guide to anyone who has
> time to dig into the issue.
>
> I did a lot of work on SWIG's ruby support, so have a good feeling for
> the various memory issues that can arise.  But I know nothing about
> libxml's api or memory management, or how the ruby bindings interact
> with it.
>

Basically, the problem is with the way we're handling libxml's structs.  
The best example is with the nodes: When freeing a node (or a document)  
libxml can free all the children, or not. With the first option we get  
problems when Ruby-side XML::Nodes have been created, since they free  
their node when garbage collected. With the second option, we'd have to  
create Node objects for every node, or otherwise ensure they get freed at  
the appropriate time - which doesn't work too well with the speed and  
efficiency thing (esp. with large documents).

Currently this is worked around with a sort of pointer-node system - when  
GCing a node, the libxml struct is only freed if it's a pointer node. IIRC  
new nodes are created as pointers, and only made into full nodes when  
they're copied outside of the document, or otherwise made independent of  
their containing document.

This kinda works (with nodes, at least) but is a bit messy. It gets a bit  
inconsistent when it comes to, for example, attributes though - there are  
still certain cases that cause things to get freed twice, or not at all.  
This is fixable, but difficult owing to the way the code's evolved - a  
case of having the tools, and the time, I guess. There's a fair few  
comments dotted around the code relating to these issues, though...

As an aside, would SWIG make any of this easier? I remember reading a  
comment (from Sam Ruby I think) that we should junk the current code and  
rewrite with SWIG. I disagreed at the time, but maybe he had a point?

> Also, does the patch from open street map fix the issue?

I honestly don't know.

>
>> The Windows issue matters, too, but since I've been lucky enough to  
>> avoid
>> Windows almost entirely in my professional life for several years now I
>> don't really know the platform well enough to handle this, and frankly  
>> I'm
>> unwilling to take the time to learn for the sake of compatibility on one
>> project. Another thing that needs to be done.
>
> I can solve the windows issues.  I have a patch to fix up memory
> allocation on windows, and with my experience with ruby-prof, I can
> easily create a win32 gem.
>

Cool - that would definitely be a big step forward for the project.

To be honest, this was one of the major things that pissed me off before -  
a lot of people were bemoaning our lack of Win32 gems, but no-one wanted  
to help with it. I ended up spending far more time that I wanted to  
messing about with a QEMU windows installation, and in the end admitted  
defeat since every way I turned seemed to require spending (more) money on  
Microsoftware I personally don't need.

Cheers,
Ross

-- 
Ross Bamford - rosco at roscopeco.co.uk

From rosco at roscopeco.co.uk  Mon Jul 23 06:34:20 2007
From: rosco at roscopeco.co.uk (Ross Bamford)
Date: Mon, 23 Jul 2007 11:34:20 +0100
Subject: [libxml-devel] Use xfree not free
In-Reply-To: <4b6f054f0707221718i233bf7b0va590acaffd6f9ca9@mail.gmail.com>
References: <46928BBC.5070502@savagexi.com>
	<4b6f054f0707111253l20242632s19582bde19328261@mail.gmail.com>
	<469535BC.9030602@savagexi.com>
	<4b6f054f0707111301x61b0b6dard9a79e2270957b40@mail.gmail.com>
	<46A3C5F3.6080309@savagexi.com>
	<4b6f054f0707221718i233bf7b0va590acaffd6f9ca9@mail.gmail.com>
Message-ID: <op.tvwybibnkc9f7k@localhost.localdomain>

On Mon, 23 Jul 2007 01:18:17 +0100, TRANS <transfire at gmail.com> wrote:

> On 7/22/07, Charlie Savage <cfis at savagexi.com> wrote:
>> Hi TRANS,
>>
>> Were you still planning on doing this?  If so, I'll patch ruby-prof to
>> work better on Windows.  And I can package up a windows gem.
>
> Yes, but right now I have to work through Red tape. Seems Ross removed
> himself from teh members list, and the only person with admin
> privliges is Sean, who once again can't be found. So I'm talking to
> Tom Copeland to see if I can get added as admin.

Hmm, I don't actually recall removing myself, but I guess I must have done  
somewhere along the line. I _was_ pretty annoyed with various things at  
the time. :)

Anyway, sorry if that caused extra work, Trans...

-- 
Ross Bamford - rosco at roscopeco.co.uk

From danj at 3skel.com  Wed Jul 25 20:55:22 2007
From: danj at 3skel.com (Dan Janowski)
Date: Wed, 25 Jul 2007 20:55:22 -0400
Subject: [libxml-devel] Maintainer
In-Reply-To: <4b6f054f0707121003v210d7b10sa49519717c70f73f@mail.gmail.com>
References: <4b6f054f0707121003v210d7b10sa49519717c70f73f@mail.gmail.com>
Message-ID: <0383D7F9-D776-4973-A029-C6CFD6A74A3D@3skel.com>

I have been on vacation. My contributions to date have been minor,  
but I am have a vested interest in getting the memory model problem  
resolved. I did something similar for the Berkeley DB library, which  
had a completely faulty binding approach and rewrote it entirely and  
now is completely memory stable and essentially fault-free.

My biggest problem is the documentation for libxml2 seems to rather  
suck in relation to the complexity of the interface. So I have not  
been able to get my head around the options for solving this problem.  
My past (long ago) uses of SWiG for other purposes left much to be  
desired, so I have no opinion about that.

The first thought I have is to look for another language binding that  
works well and see if that approach can be made to work for Ruby.

This does seem to be the best XML solution, so getting it all fixed  
up seems like a really good idea. It seems to be bigger than one  
person can manage on a part time basis, but there needs to be a  
unified vision of how to approach the problem.

Dan

On Jul 12, 2007, at 13:03, TRANS wrote:

> I see that Ross Bramford is not even listed as a project member any
> more --let alone the maintainer. So that means things fall back to
> Sean Chittenden -- but is he around? I see three other members listed:
>
>   Dan Janowski
>   Laurent Sansonetti
>   Pat Eyler
>
> Are any of you guys around?
>
> I fear there's no one at the hem again, but there are still important
> issues that need to be addressed. From the looks of it there hasn't
> been a commit since January, though there have been patches offered up
> on this list since then.
>
> What's the word?
>
> T.
> _______________________________________________
> libxml-devel mailing list
> libxml-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/libxml-devel


From cfis at savagexi.com  Wed Jul 25 23:58:37 2007
From: cfis at savagexi.com (Charlie Savage)
Date: Wed, 25 Jul 2007 21:58:37 -0600
Subject: [libxml-devel] Maintainer
In-Reply-To: <0383D7F9-D776-4973-A029-C6CFD6A74A3D@3skel.com>
References: <4b6f054f0707121003v210d7b10sa49519717c70f73f@mail.gmail.com>
	<0383D7F9-D776-4973-A029-C6CFD6A74A3D@3skel.com>
Message-ID: <46A81BED.5050700@savagexi.com>

> My biggest problem is the documentation for libxml2 seems to rather  
> suck in relation to the complexity of the interface. So I have not  
> been able to get my head around the options for solving this problem.  
> My past (long ago) uses of SWiG for other purposes left much to be  
> desired, so I have no opinion about that.

SWIG is fine - but it isn't a magic bullet that will solve the issue. 
It does offer once nice feature, you can set it up so that it returns 
the same Ruby object for a given C object (as opposed to having 4 ruby 
objects all point to the same C object).  But SWIG is not the easiest 
thing to use, and if the bindings already exist anyway, I'd probably 
just stick with them (unless they cover such a small part of the api 
that it would be worth autogenerating the rest).

> 
> The first thought I have is to look for another language binding that  
> works well and see if that approach can be made to work for Ruby.

Maybe Python?  Although it uses reference counting, not a GC, so maybe 
not appropriate.

Charlie
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3237 bytes
Desc: S/MIME Cryptographic Signature
Url : http://rubyforge.org/pipermail/libxml-devel/attachments/20070725/fde42ccf/attachment.bin 

From hello at timperrett.com  Thu Jul 26 10:23:39 2007
From: hello at timperrett.com (Tim Perrett)
Date: Thu, 26 Jul 2007 15:23:39 +0100
Subject: [libxml-devel] Malloc problem when using Schema.new()
Message-ID: <45A4C7C9-CD6D-4655-9BF6-B28901366316@timperrett.com>

Hey all

My apologies for posting this to the devel mailing list, but there  
didnt seem to be any other type of list for libxml.
I have a schema that I want to read it using XML::Schema.new('path/to/ 
file.xsd') however when I do that with this particular schema it eats  
about 2GB of RAM and 2.5GB of virtual memory on my box before it  
encounters a malloc exception.

I was wondering if there was either a way to stream read the schema  
(as it is large, 33,601 lines) or could someone explain to me whats  
going on under the hood so I can perhaps do some kind of workaround?  
The schema in question is http://www.cip4.org/Schema/ 
JDFSchema_1_3_20070212.zip

Any help would be most appreciated.

Cheers

Tim

From transfire at gmail.com  Thu Jul 26 19:53:04 2007
From: transfire at gmail.com (TRANS)
Date: Thu, 26 Jul 2007 16:53:04 -0700
Subject: [libxml-devel] Malloc problem when using Schema.new()
In-Reply-To: <45A4C7C9-CD6D-4655-9BF6-B28901366316@timperrett.com>
References: <45A4C7C9-CD6D-4655-9BF6-B28901366316@timperrett.com>
Message-ID: <4b6f054f0707261653t99f0b0fma3535cf0e75fd729@mail.gmail.com>

On 7/26/07, Tim Perrett <hello at timperrett.com> wrote:
> Hey all
>
> My apologies for posting this to the devel mailing list, but there
> didnt seem to be any other type of list for libxml.
> I have a schema that I want to read it using XML::Schema.new('path/to/
> file.xsd') however when I do that with this particular schema it eats
> about 2GB of RAM and 2.5GB of virtual memory on my box before it
> encounters a malloc exception.
>
> I was wondering if there was either a way to stream read the schema
> (as it is large, 33,601 lines) or could someone explain to me whats
> going on under the hood so I can perhaps do some kind of workaround?
> The schema in question is http://www.cip4.org/Schema/
> JDFSchema_1_3_20070212.zip
>
> Any help would be most appreciated.

I'd like to know how much is libxml2 and how much is the ruby binding.
have you/can you try if in C?

T.

From hello at timperrett.com  Mon Jul 30 04:04:23 2007
From: hello at timperrett.com (Tim Perrett)
Date: Mon, 30 Jul 2007 09:04:23 +0100
Subject: [libxml-devel] Malloc problem when using Schema.new()
In-Reply-To: <4b6f054f0707261653t99f0b0fma3535cf0e75fd729@mail.gmail.com>
References: <45A4C7C9-CD6D-4655-9BF6-B28901366316@timperrett.com>
	<4b6f054f0707261653t99f0b0fma3535cf0e75fd729@mail.gmail.com>
Message-ID: <B0F9DA13-E363-4B98-AB01-C786E553F95C@timperrett.com>

Thats interesting, so you think the issue lay with ruby and not  
libxml? Ive done stuff with cocoa on OSX, but not straight C. Any  
pointers how I might be able to test if its the ruby binding? I am  
afraid im not familiar with straight C. I've done stuff with  
objective-c on OSX however if that helps?

Many thanks for the reply

Tim


On 27 Jul 2007, at 00:53, TRANS wrote:

> I'd like to know how much is libxml2 and how much is the ruby binding.
> have you/can you try if in C?
>
> T.


From transfire at gmail.com  Mon Jul 30 04:19:20 2007
From: transfire at gmail.com (TRANS)
Date: Mon, 30 Jul 2007 01:19:20 -0700
Subject: [libxml-devel] Malloc problem when using Schema.new()
In-Reply-To: <B0F9DA13-E363-4B98-AB01-C786E553F95C@timperrett.com>
References: <45A4C7C9-CD6D-4655-9BF6-B28901366316@timperrett.com>
	<4b6f054f0707261653t99f0b0fma3535cf0e75fd729@mail.gmail.com>
	<B0F9DA13-E363-4B98-AB01-C786E553F95C@timperrett.com>
Message-ID: <4b6f054f0707300119w3c704565g91ef69b099e4e703@mail.gmail.com>

On 7/30/07, Tim Perrett <hello at timperrett.com> wrote:
> Thats interesting, so you think the issue lay with ruby and not
> libxml?

Possibly, though I suspect the bindings themselves more. But I have no
sure idea, which is why I'm wondering.

> Ive done stuff with cocoa on OSX, but not straight C. Any
> pointers how I might be able to test if its the ruby binding? I am
> afraid im not familiar with straight C. I've done stuff with
> objective-c on OSX however if that helps?

Could try compiling some examples:

  http://xmlsoft.org/examples/index.html

Then perhaps modify and create a ruby version to compare,

T.

From hello at timperrett.com  Mon Jul 30 04:59:52 2007
From: hello at timperrett.com (Tim Perrett)
Date: Mon, 30 Jul 2007 09:59:52 +0100
Subject: [libxml-devel] Malloc problem when using Schema.new()
In-Reply-To: <4b6f054f0707300119w3c704565g91ef69b099e4e703@mail.gmail.com>
References: <45A4C7C9-CD6D-4655-9BF6-B28901366316@timperrett.com>
	<4b6f054f0707261653t99f0b0fma3535cf0e75fd729@mail.gmail.com>
	<B0F9DA13-E363-4B98-AB01-C786E553F95C@timperrett.com>
	<4b6f054f0707300119w3c704565g91ef69b099e4e703@mail.gmail.com>
Message-ID: <1E027A34-A69F-4562-B5D1-DBD8357F37DA@timperrett.com>

In terms of whats going on under the hood with Schema.new()... XSD is  
obviously just xml so I would imagine its parsed as per any other  
file? Is there anything additional that goes on when you call  
Schema.new?

Also, I have had other people tell me that the problem is due to  
libxml-ruby trying to load the xsd with a DOM method, and hence it  
uses so much memory. Is it even possible to use a streaming method  
and still be able to validate XSI?

Cheers

Tim


On 30 Jul 2007, at 09:19, TRANS wrote:

> Could try compiling some examples:
>
>   http://xmlsoft.org/examples/index.html
>
> Then perhaps modify and create a ruby version to compare,
>
> T.