[Nitro] Og threaded mode leak.

Jonathan Buch john at oxyliquit.de
Sat Feb 24 08:40:00 EST 2007


Hi,

> Jonathan told me on the #irc that he has found a memory leak when running Og
> in multithreaded mode (according to him this was an old bug, never fixed). I
> would like to ask Jonathan (or someone else) to explain this bug to me in
> detail so that I can fix it.


the code in question of manager.rb:

def get_store:

unless st = thread[:og_store] and st.is_a?(@store_class)
     if 0 == @pool.size
         initialize_store
     end
     st = @pool.pop
     thread[:og_store] = st
end

This reads 'get stores, if it's zero, just make new stores'.  This is one
part of the problem.
The second part is, that when you remove that 'make indefinit stores' you
get a second problem, which is 'stalling threads' because the og-stores
are not being forced to return to the pool.

The fix is to make get_store like this:

def store
   yield get_store
   put_store
end

And always go through the block so the stores are returned at once so
threads are not 'blocking the application to death'.

Test case (original by [Alex], revised heavily by me)
http://pastie.caboo.se/42635

What it does, is first creating 1500 entries in a table, and after that
it tries (in 1500 threads) to pull those entries out again.  It counts
the errors it gets, which should be 0.

The behaviour I'm getting with psql:
"FATAL:  sorry, too many clients already"  (after configured limit of
50 concurrent connections) and 1453 failing threads.  This is only
a testcase, but have been getting that with oxyliquit as well, it just
leaks stores.  (try doing `ps ax | grep postgres`, it's funny :P)

Alex originally had the problem with sqlite, that everything just broke
down after a few threads more.  This had to do with file-descriptors
and that it openened the sqlite db too often.  I originally didn't see
the problem (because everything worked fine with psql).  Now I know that
there isn't a problem with FCGI (because it sets Og.thread_safe to false)
and so doesn't use the pool at all.

Alex made a patch once, implementing the idea above, only allowing
access to the store via a block, so it always is returned.  It sadly
didn't get the attention it deserved (partly my fault).  The changes
to be done aren't really hard, there are just many of the .get_store
and .store which all have to be wrapped.

I hope that helps,

Jo

-- 
Feel the love
http://pinkjuice.com/pics/ruby.png


More information about the Nitro-general mailing list