<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Forgive me for not having read the whole thread, however, there is one thing that seems to be really important, and that is, ruby hardly ever runs the damned GC. It certainly doesn't do full runs nearly often enough (IMO).<div><br></div><div>Also, implicit OOMEs or GC runs quite often DO NOT affect the extensions correctly. I don't know what rmagick is doing under the hood in this area, but having been generating large portions of country maps with it (and moving away from it very rapidly), I know the GC doesn't do "The Right Thing".</div><div><br></div><div>First call of address is GC_MALLOC_LIMIT and friends. For any small script that doesn't breach that value, the GC simply doesn't run. More than this, RMagick, in it's apparent 'wisdom' never frees memory if the GC never runs. Seriously, check it out. Make a tiny script, and make a huge image with it. Hell, make 20, get an OOME, and watch for a run of the GC. The OOME will reach your code before the GC calls on RMagick to free.</div><div><br></div><div>Now, add a call to GC.start, and no OOME. Despite the limitations of it (ruby performance only IMO), most of the above experience was built up on windows, and last usage was about 6 months ago, FYI.</div><div><br></div><div><div>On 24 Mar 2008, at 20:37, Luis Lavena wrote:<br><blockquote type="cite">On Mon, Mar 24, 2008 at 4:59 PM, Scott Windsor <<a href="mailto:swindsor@gmail.com">swindsor@gmail.com</a>> wrote:<br><blockquote type="cite">On Mon, Mar 24, 2008 at 12:18 PM, Luis Lavena <<a href="mailto:luislavena@gmail.com">luislavena@gmail.com</a>> wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">On Mon, Mar 24, 2008 at 3:58 PM, Scott Windsor <<a href="mailto:swindsor@gmail.com">swindsor@gmail.com</a>> wrote:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">You're using *RMagick*, not ImageMagick directly. If you used the<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">later (via system calls) there will no be memory leakage you can worry<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">about.<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">You're correct - I'm using 'RMagick' - and it uses a large amount of memory.<br></blockquote><blockquote type="cite">But that's not really the overall point. My overall point is how to<br></blockquote><blockquote type="cite">properly handle a rails app that uses a great deal of memory during each<br></blockquote><blockquote type="cite">request. I'm pretty sure this happens in other rails applications that<br></blockquote><blockquote type="cite">don't happen to use 'RMagick'.</blockquote></blockquote><div><br></div><div>Personally, I'll simply say call the GC more often. Seriously. I mean it. It's not *that* slow, not at all. In fact, I call GC.start explicitly inside of by ubygems.rb due to stuff I have observed before:</div><div><br></div><div><a href="http://blog.ra66i.org/archives/informatics/2007/10/05/calling-on-the-gc-after-rubygems/">http://blog.ra66i.org/archives/informatics/2007/10/05/calling-on-the-gc-after-rubygems/</a> - N.B. This isn't "FIXED" it's still a good idea (gem 1.0.1).</div><div><a href="http://zdavatz.wordpress.com/2007/07/18/heap-fragmentation-in-a-long-running-ruby-process/">http://zdavatz.wordpress.com/2007/07/18/heap-fragmentation-in-a-long-running-ruby-process/</a></div><div><br></div><div>Now, by my reckoning (and a few production apps seem to be showing emperically (purely emperical, sorry)) we should be calling on the GC whilst loading up the apps. I mean come on, when are a really serious number of temporary objects being created. Actually, it's when rubygems loads, and that's the first thing that happens in, hmm, probably over 90% of ruby processes out there.</div><div><br></div><blockquote type="cite"><br>Yes, I faced huge memory usage issues with other things non related to<br>image processing and found that a good thing was move them out of the<br>request-response cycle and into a out-of-bound background job.<br><br><blockquote type="cite"><br></blockquote><blockquote type="cite">So far, running the GC under fastcgi has given me pretty good results. The<br></blockquote><blockquote type="cite">zombing issue with fast cgi is a known issue with mod_fastcgi and I'm pretty<br></blockquote><blockquote type="cite">sure unrelated to RMagick or garbage collection.<br></blockquote><blockquote type="cite"><br></blockquote><br>Yes, but even you "reclaim" the memory with GC, there will be pieces<br>that wouldn't be GC'ed ever, since the leaked in the C side, outside<br>GC control (some of the RMagick and ImageMagick mysteries).</blockquote><div><br></div><div>Sure, but leaks are odd things. Some processes that appear to be leaking are really just fragmenting (allocating more ram due to lack of 'usable' space on 'the heap'. Call the GC more often, take a 0.01% performance hit, and monitor. I bet it'll get better. In fact, you can drop fragmentation the first allocated segment significantly just by calling GC.start after a rubygems load, if you have more than a few gems.</div><br><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><font class="Apple-style-span" color="#006312"><br></font>Can you tell me how you addressed the "schedule" of the garbage<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">collection execution on your previous scenario? AFAIK most of the<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">frameworks or servers don't impose to the user how often GC should be<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">performed.</blockquote></blockquote></blockquote><div><br></div><div>In fact there are many rubyists who hate the idea of splatting GC.start into processes. Given what I've seen, I'm willing to reject that notion completely. Test yourself, YMMV.</div><div><br></div><div>FYI, even on windows under the OCI, where performance for the interpreter sucks, really really hard, I couldn't reliably measure the runtime of a call to GC.start after loading rubygems. I don't know what kind of 'performance' people are after, but I can't see the point in not running the GC more often, especially for 'more common' daemon load. Furthermore, hitting the kernel for more allocations more often, is actually pretty slow too, so this may actually even result in faster processes under *certain* conditions.</div><div><br></div><div>Running a lib like RMagick, I would say you *should* be doing this, straight up, no arguments.</div><br><blockquote type="cite"><blockquote type="cite"><br></blockquote><blockquote type="cite">In the previous scenario I was using fast_cgi with rails. In my previous<br></blockquote><blockquote type="cite">reply I provided a link to the rails fastcgi dispatcher.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><a href="http://dev.rubyonrails.org/browser/trunk/railties/dispatches/dispatch.fcgi">http://dev.rubyonrails.org/browser/trunk/railties/dispatches/dispatch.fcgi</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">In addtion, in other languages and other language web frameworks there are<br></blockquote><blockquote type="cite">provisions to control garbage collection (for languages that have garbage<br></blockquote><blockquote type="cite">collections, of course).<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">I'll bet is rails specific, or you should take a look at the fcgi ruby<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">extension, since it is responsible, ruby-side, of bridging both<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">worlds.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">This is done in the Rails FastCGI dispatcher. I believe that the equivalent<br></blockquote><blockquote type="cite">of this in Mongrel is the Mongrel Rails dispatcher. Since the Mongrel Rails<br></blockquote><blockquote type="cite">dispatcher is distributed as a part of Mongrel, I'd say this code is owned<br></blockquote><blockquote type="cite">by Mongrel, which bridges these two worlds when using mongrel as a<br></blockquote><blockquote type="cite">webserver.</blockquote></blockquote><div><br></div><div>It doesn't *really* matter where you run the GC. It matters that it runs, how often, and what it's doing. If you're actually calling on the GC and freeing nothing, that's stupid, but if you've run RMagick up, just call GC.start anyway, and I'm pretty sure it'll help. There's certainly no harm in investigating this, unless you're doing something silly with weakrefs.</div><div><br></div><br><blockquote type="cite">Then you could provide a different Mongrel Handler that could perform<br>that, or even a series of GemPlugins that provide a gc:start instead<br>of plain 'start' command mongrel_rails scripts provides.</blockquote><div><br></div><div><br></div><div>$occasional_gc_run_counter = 0</div><div>before_filter :occasional_gc_run</div><div><br></div><div>def occasional_gc_run</div><div> $occasional_gc_run_counter += 1</div><div> if $occasional_gc_run_counter > 1_000</div><div> $occasional_gc_run_counter = 0</div><div> GC.start</div><div> end</div><div>end</div><div><br></div><div>Or whatever. It doesn't really matter that much where you do this, or when, it just needs to happen every now and then. More importantly, add a GC.start to the end of environment.rb, and you will have literally half the number of objects in ObjectSpace.</div><div><br></div><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">On a personal note, I believe is not responsibility of Mongrel, as a<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">webserver, take care of the garbage collection and leakage issues of<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">the Vm on which your application runs. In any case, the GC of the VM<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">(MRI Ruby) should be enhanced to work better with heavy load and long<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">running environments.</blockquote></blockquote></blockquote><div><br></div>Right, and it's not just the interpreter, although indirection around this stuff can help. (such as compacting).</div><div><br><blockquote type="cite"><blockquote type="cite"><br></blockquote><blockquote type="cite">Ruby provides an API to access and call the Garbage Collector. This<br></blockquote><blockquote type="cite">provides ruby application developers the ability to control when the garbage<br></blockquote><blockquote type="cite">collection is run because in some cases, there may be an<br></blockquote><blockquote type="cite">application-specific reason to prevent or explicity run the GC. Web servers<br></blockquote><blockquote type="cite">are a good example of these applications where state may help determine a<br></blockquote><blockquote type="cite">better time to run the GC. As you're serving each request, you're generally<br></blockquote><blockquote type="cite">allocating a number of objects, then rendering output, then moving on to the<br></blockquote><blockquote type="cite">next request.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">By limiting the GC to run in between requests rather than during requests<br></blockquote><blockquote type="cite">you are trading request time for latency between requests. This is a<br></blockquote><blockquote type="cite">trade-off that I think web application developers should deciede, but by no<br></blockquote><blockquote type="cite">means should this be a default or silver bullet for all. My position is<br></blockquote><blockquote type="cite">that this just be an option within Mongrel as a web server.<br></blockquote><blockquote type="cite"><br></blockquote></blockquote><div><br></div>Right, I think this is important too. You're absolutely right that there's no specific place to provide a generic solution. In rails the answer may be simple, but that's because rails outer architecture is simplistic. No threads, no out-of-request processing, and so on.<br><blockquote type="cite"><br>--gc-interval maybe?<br><br>Now that you convinced me and proved your point, having the option to<br>perform it (optionally, not forced) will be something good to have.</blockquote><div><br></div><div>Surely you can just:</div><div><br></div><div>require 'thread'</div><div>Thread.new { loop { sleep GC_FORCE_INTERVAL; GC.start } }</div><div><br></div><div>In environment.rb in that case.</div><div><br></div><div>Of course, this is going to kill performance under evented_mongrel, thin and so on. I'd stay away from threaded solutions. _why blogged years ago about the GC, trying to remind people that we actually have control. I know ruby is supposed to abstract memory problems etc away from us, and for the most part it does, but hey, no one's perfect, right? :-)</div><div><br></div><a href="http://whytheluckystiff.net/articles/theFullyUpturnedBin.html">http://whytheluckystiff.net/articles/theFullyUpturnedBin.html</a></div><div><br><blockquote type="cite">Patches are Welcome ;-)</blockquote><div><br></div>Have fun! :o)</div><div><br></div></div></body></html>