[raleigh.rb] Enumerable#group_by is gobbling up memory

[John W. Long]

Have you considered Rails 2.3's find_in_batches? It lets you do stuff  
like this:

   Customer.find_in_batches(:conditions => {:active => true}) do | 
customer_group|
     customer_group.each { |customer| customer.update_account_balance! }
   end

More here:

http://guides.rubyonrails.org/2_3_release_notes.html#batch-processing

--
John Long
http://wiseheartdesign.com

On Mar 10, 2009, at 10:00 AM, Matthew Bass wrote:

> Has anyone else experienced problems with Rails' Enumerable#group_by  
> method hogging memory when dealing with large arrays of ActiveRecord  
> objects?
>
> I have a view in my app that uses group_by on a collection of AR  
> objects (5600+). When I hit this view, Rails spins for a good minute  
> or so and the memory usage of the Ruby process climbs from 55 MB to  
> peak out around 650 MB. I don't see anything obvious in the  
> implementation of group_by that would explain this phenomenon:
>
> def group_by
>   inject({}) do |groups, element|
>     (groups[yield(element)] ||= []) << element
>   groups
> end
>
> I tried re-implementing group_by and clearing the array when  
> finished, to no avail:
>
>   def group_by(array, key)
>     groups = {}
>     array.each { |e| (groups[e.send(key)] ||= []) << e }
>     array.clear
>     groups
> end
>
> Setting the array to nil and calling GC.start didn't help either.
>
> Anyone have suggestions as to how I could refactor group_by to avoid  
> this memory problem? It looks to me like the hashing process itself  
> is what's gobbling up the memory, but I could be wrong. Every time I  
> hit the page, the memory usage increases. (Letting the database  
> handle the grouping is *not* an option at this point.)
>
> Matthew
> _______________________________________________
> raleigh-rb-members mailing list
> raleigh-rb-members at rubyforge.org
> http://rubyforge.org/mailman/listinfo/raleigh-rb-members