[Nitro] Og threaded mode leak.

George Moschovitis george.moschovitis at gmail.com
Sun Feb 25 11:32:33 EST 2007


Jonathan, thanks for the detailed description. I will have a look at this.
Btw, the get_store method is altered, this is not my initial implementation.
Nitro would restore the store after each request (the last command in the
request handling loop in adapters). Some patch must have altered the code.
In any case I will have a look at this.

thanks,
George.

On 2/24/07, Jonathan Buch <john at oxyliquit.de> wrote:
>
> Hi,
>
> > Jonathan told me on the #irc that he has found a memory leak when
> running Og
> > in multithreaded mode (according to him this was an old bug, never
> fixed). I
> > would like to ask Jonathan (or someone else) to explain this bug to me
> in
> > detail so that I can fix it.
>
>
> the code in question of manager.rb:
>
> def get_store:
>
> unless st = thread[:og_store] and st.is_a?(@store_class)
>      if 0 == @pool.size
>          initialize_store
>      end
>      st = @pool.pop
>      thread[:og_store] = st
> end
>
> This reads 'get stores, if it's zero, just make new stores'.  This is one
> part of the problem.
> The second part is, that when you remove that 'make indefinit stores' you
> get a second problem, which is 'stalling threads' because the og-stores
> are not being forced to return to the pool.
>
> The fix is to make get_store like this:
>
> def store
>    yield get_store
>    put_store
> end
>
> And always go through the block so the stores are returned at once so
> threads are not 'blocking the application to death'.
>
> Test case (original by [Alex], revised heavily by me)
> http://pastie.caboo.se/42635
>
> What it does, is first creating 1500 entries in a table, and after that
> it tries (in 1500 threads) to pull those entries out again.  It counts
> the errors it gets, which should be 0.
>
> The behaviour I'm getting with psql:
> "FATAL:  sorry, too many clients already"  (after configured limit of
> 50 concurrent connections) and 1453 failing threads.  This is only
> a testcase, but have been getting that with oxyliquit as well, it just
> leaks stores.  (try doing `ps ax | grep postgres`, it's funny :P)
>
> Alex originally had the problem with sqlite, that everything just broke
> down after a few threads more.  This had to do with file-descriptors
> and that it openened the sqlite db too often.  I originally didn't see
> the problem (because everything worked fine with psql).  Now I know that
> there isn't a problem with FCGI (because it sets Og.thread_safe to false)
> and so doesn't use the pool at all.
>
> Alex made a patch once, implementing the idea above, only allowing
> access to the store via a block, so it always is returned.  It sadly
> didn't get the attention it deserved (partly my fault).  The changes
> to be done aren't really hard, there are just many of the .get_store
> and .store which all have to be wrapped.
>
> I hope that helps,
>
> Jo
>
> --
> Feel the love
> http://pinkjuice.com/pics/ruby.png
> _______________________________________________
> Nitro-general mailing list
> Nitro-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/nitro-general
>



-- 
http://blog.gmosx.com
http://cull.gr
http://www.joy.gr
http://nitroproject.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/nitro-general/attachments/20070225/bdc8bb96/attachment.html 


More information about the Nitro-general mailing list