<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
The one thing I didn't include in the post is the output of "vmmap" on
OS X prior to this change.<br>
<br>
It showed large amounts of memory both marked as "(freed)" and some
not. That pattern reminded me of internal fragmentation in a malloc
implementation, which is a standard problem to be overcome when
implementing one. If anyone is interested, I could post that as well.<br>
<br>
<div class="moz-signature">
<div style="background-color: white; color: black;"> <span
class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span
class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span
class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span
class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
<div>
<p style="margin: 0px;"><font class="Apple-style-span" size="6"><span
class="Apple-style-span" style="font-size: 19px;"><b>Joe Khoobyar<br>
</b></span></font></p>
<p style="margin: 0px;"><font class="Apple-style-span" color="#565757">Chief
Technical Officer & Lead Developer</font></p>
<p style="margin: 0px 0px 12px;"><span class="Apple-style-span"
style="color: rgb(47, 82, 16); font-size: 19px; font-weight: bold;">CollectiveX</span>
<font
style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; font-size: 14px; line-height: normal; font-size-adjust: none; font-stretch: normal; color: rgb(0, 119, 68);"
color="#007744" face="Helvetica" size="4"> - <i>Groups that work!</i></font></p>
<p style="margin: 0px;"><font
style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: normal; font-size-adjust: none; font-stretch: normal; color: rgb(86, 87, 87);"
color="#565757" face="Helvetica" size="3">mobile: 585.245.2902 </font></p>
<p style="margin: 0px;"><font
style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: normal; font-size-adjust: none; font-stretch: normal; color: rgb(86, 87, 87);"
color="#565757" face="Helvetica" size="3">email: </font><font
style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: normal; font-size-adjust: none; font-stretch: normal; color: rgb(0, 31, 230);"
color="#001fe6" face="Helvetica" size="3"><a
href="mailto:joe@collectivex.com">joe@collectivex.com</a></font></p>
<p style="margin: 0px;"><font
style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: normal; font-size-adjust: none; font-stretch: normal; color: rgb(86, 87, 87);"
color="#565757" face="Helvetica" size="3">web: </font><font
style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: normal; font-size-adjust: none; font-stretch: normal; color: rgb(0, 31, 230);"
color="#001fe6" face="Helvetica" size="3"><a
href="http://www.collectivex.com/">www.collectivex.com</a> <a
href="http://www.groupsites.com/">www.groupsites.com</a></font><br>
<br>
<font
style="font-family: Helvetica; font-style: italic; font-variant: normal; font-weight: normal; font-size: 11px; line-height: normal; font-size-adjust: none; font-stretch: normal; color: rgb(108, 108, 108);"
color="#565757" face="Helvetica" size="3"> The third-rate mind is only
happy when it is thinking with the majority.<br>
The second-rate mind is only happy when it is thinking with the
minority.<br>
The first-rate mind is only happy when it is thinking.<br>
<br>
— A.A. Milne</font></p>
</div>
</span></span></span></span> </div>
</div>
<br>
<br>
Charlie Savage wrote:
<blockquote cite="mid:4990E58E.5010103@savagexi.com" type="cite">Hey
Joe,
<br>
<br>
<blockquote type="cite">Revision 783 in subversion applies this
change. With this change, test08.rb holds about 13 MB on a Mac, where
it held over 120 MB before.
<br>
</blockquote>
<br>
Ah, interesting. It is amazing how much memory libxml seems to grab on
simple test cases.
<br>
<br>
<blockquote type="cite">We are simply using libxml2's memory
management hooks to direct it's alloc/free functions and related
functions to ruby's versions of those functions. What I originally
planned (which is why I'm using xmlGcMemSetup) was to direct what
libxml2 calls "atomic" memory allocations to the version of the ruby's
allocator which is able to run the garbage collector. </blockquote>
<br>
> Basically what this does is have both libxml2 and ruby use the
same
<br>
> allocator and allow ruby to run it's GC even in response to
libxml2
<br>
> allocations which keeps memory usage down much more easily. I'm
already
<br>
> running this on all of our production servers and it yielded
instant
<br>
> benefits.
<br>
<br>
What I gather from this is that somehow libxml is not freeing memory
that it allocates. Except that memory is freed, albeit indirectly,
when Ruby when it runs a garbage collection freeing Ruby objects and
then the underlying libxml objects. And from using valgrind, I haven't
seen any large memory leaks in libxml-ruby for quite a while.
<br>
<br>
So I'm wondering what is going on. Does libxml just allocate a big
chunk of memory at startup, and then allocates/frees from that memory
over the period that a process runs? And by telling libxml to use
Ruby's allocator, then that issue goes away? If that's the case, is
there some setting to libxml to set the starting memory. Or is there
something else entirely different going on?
<br>
<br>
And is there any downside to having libxml get memory from Ruby?
Something along the lines that ruby's memory allocation wouldn't be as
efficient for libxml's usage patterns?
<br>
<br>
I'd just like to understand this a bit better since its such a big
change (not code wise, but running wise). Should we hedge our bets and
make it settable somehow - is that even possible?
<br>
<br>
<blockquote type="cite">Just wondering: does anyone have any
feedback on this change or on this subject? FYI: I've been running
it on 32-bit ruby on OS X Leopard and on several 64-bit UNIX instances
for half a week now.
<br>
</blockquote>
<br>
Yeah, I'd have to give it a try on Windows. Theoretically, it sounds
fine, but we'll have to see what really happens.
<br>
<br>
Anyway, excellent work to figure this out!
<br>
<br>
Charlie
<br>
<br>
</blockquote>
</body>
</html>