Forums | Admin

Discussion Forums: help

Start New Thread Start New Thread

 

By: Masahiro Tanaka
RE: Memory Allocation problems [ reply ]  
2010-03-20 12:24
NArray includes garbage collection mechanism. It can be confirmed by the following code.

require 'narray'

#GC.disable

100.times do
a=NArray.float(1000000)
end

system "ps u -p #{Process.pid}"

The result is:
$ ruby test.rb
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
masa 19965 0.0 0.9 37964 18864 pts/4 S+ 21:04 0:00 ruby test.rb

If `#' is removed for GC.disable to be valid:
$ ruby test.rb
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
masa 19969 68.0 38.6 803884 784840 pts/4 S+ 21:04 0:00 ruby test.rb

The description you cited is specific to RMagick, so it does not applicable to NArray.

By: Christoffer LervÄg
Memory Allocation problems [ reply ]  
2010-03-19 20:06
First of all, thanks for your excellent work with this gem!

I've started on a project where I read hundreds of files into memory, then cycle through each file, processing it in NArrays and eventually writes the resulting data back to files. Somewhere through this process, I unfortunately run out of memory and the program halts. The size of these data files is not bigger than what should be able to fit in memory if garbage collection is working properly.

After a bit of searching I have found a discussion debating what I think is a plausible explanation for this problem. Im pasting the relevant part of that discussion at the end of this message.

It seems to me that NArray does not have a destroy! method to tell the C library to clean up the memory itself. Is this something you could add to NArray?

Link to RMagick destroy! method documentation:
http://studio.imagemagick.org/RMagick/doc/image1.html#destroy_bang


A copy of a previous discussion of a similar problem:
-------------------------------------------------
Beware of C extensions which allocate large chunks of memory themselves.

As an example, when you load an image using RMagick, the entire bitmap gets loaded into memory inside the ruby process. This may be 30 meg or so depending on the size of the image.
However, most of this memory has been allocated by RMagick itself. All ruby knows about is a wrapper object, which is tiny(1).
Ruby only thinks it's holding onto a tiny amount of memory, so it won't bother running the GC. In actual fact it's holding onto 30 meg.
If you loop over a say 10 images, you can run yourself out of memory really fast.

The preferred solution is to manually tell the C library to clean up the memory itself - RMagick has a destroy! method which does this. If your library doesn't however, you may need to forcibly run the GC yourself, even though this is generally discouraged.

(1): Ruby C extensions have callbacks which will get run when the ruby runtime decides to free them, so the memory will eventually be successfully freed at some point, just perhaps not soon enough.