[Backgroundrb-devel] Memory leak?

Ezra Zygmuntowicz ezmobius at gmail.com
Mon Aug 7 19:11:05 EDT 2006

On Aug 7, 2006, at 3:51 PM, Joel Hayhurst wrote:

> I am using BackgrounDRb to resize images after they are accepted into
> our system.  However, it has become clear that this creates a memory
> leak.  I'm not sure exactly where the leak exists, but I don't think
> it's in my own code as it is presently being used in production and
> has resized thousands of images without a leak occurring, thanks to
> calling GC.start after every RMagick image read.
> I have it working like this.  First of all, I changed the start
> script to include environment.rb and therefore the ENTIRE Rails
> environment, because my Image model is an engine plugin.  Then, I
> have it perform this upon acceptance:
>          MiddleMan.new_worker :class => :resize_worker, :args =>  
> self.id
> And here are the contents of ResizeWorker:
> class ResizeWorker < BackgrounDRb::Rails
>    def do_work(args)
>      image = Image.find(args)
>      image.resize_all
>      terminate
>    end
> end
> If I run this on a few dozen images simultaneously I watch as top
> starts slowly requiring more memory for backgroundrb.  Dropping into
> script/console tells me two things.
> For one, MiddleMan.jobs still has every job, despite my terminate()
> call.  Second, calling destroy_worker on any of these calls, or gc!
> (Time.now), does not free up any of the memory, even though the jobs
> disappear.  GC.start also does not free up any memory.
> If you have any advice on this, that would be great.
> Thanks,
> Joel

Hey Joel-

	Calling terminate does not immediately terminate the worker class.  
All it does is flag the thread as ok to terminate when a  
delete_worker call is made. So you will still need to call  
delete_worker or gc! in order to clear your worker out of @jobs. As  
far as actually getting the process memory back, you need to realize  
that even if you call GC.start, that doesn't mean that ruby will  
automatically clean up all garbage. What platform are you on? And can  
you confirm whether the real memory or the virtual memory decrease  
after a delete_worker and a GC.start ?

	Where did you call GC.start from in your experiment? If you called  
it from the command line then it will not effect the drb server. Try  
this, add GC.start to the gc! method in the backgroundrb.rb file in  
the plugin root and restarting your drb server. like this:

     def gc!(age)
       @timestamps.each do |job_key, timestamp|
         if timestamp[timestamp[:expire_type]] < age

	And now run a bunch of workers until memory usage grows. Then run  
MiddleMan.gc! Time.now from script/console and wait a little bit to  
see if your memory usage goes down.  Please report back if this does  
solve your problem and I will add it to the distro.

	Also RMagick is know to leak sometimes so I have been using  
MiniMagick myself when I have needed to do image resizing in a worker.

	Let me know if none of these things solve your issue and I will try  
to get to the bottom of this.


More information about the Backgroundrb-devel mailing list