[Boulder-Denver Ruby Group] send_file "memory leak"

[Tony Arcieri]

When I gave my talk on Rublique (which has now been rendered quite obsolete
by Evan Weaver's Bleak House, which uses C instrumentation much faster than
ObjectSpace) I mentioned at the end that the memory leak I was experiencing
was due to send_file.  A lot of people were confused because of that,
specifically what the nature of the memory leak was, and I didn't exactly do
a good job of describing what the problem was.  It's not really a memory
leak.  The problem is Rails returns a StringIO to Mongrel which represents
the entire content body.  That's fine for most documents, but what happens
when you're trying to send a 1GB+ file?

send_file returns a Proc object as the :text option to render.  This proc
contains a simple routine to print out the file.  The problem with the
Rails/Mongrel combination is that this Proc simply writes to a StringIO,
which when the entire content body has been emitted (i.e. the entire
contents of the file) returns it all to Mongrel, which then sends it to the
network:

70:             len = options[:buffer_size] || 4096
71:             File.open(path, 'rb') do |file|
72:               while buf = file.read(len)
73:                 output.write(buf)
74:               end
75:             end


When Mongrel is serving multiple, large files to the network, this is
obviously going to eat up a lot of RAM.

The intermediate StringIO is what causes the massive memory consumption.  If
you could instance_eval the Proc returned by send_file and expose a method
which writes directly to the network, the intermediate StringIO could be
avoided.

The advantages to this would be pure-Ruby instrumentation which can act upon
exactly how much data is sent, for example a statistics tracking system
which monitors how much of a given file was sent after it was requested.
You can do this now, but the cost is every file presently being sent will
remain in memory until the transfer ends.

-- 
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/bdrg-members/attachments/20070525/d785b1fd/attachment.html