[Backgroundrb-devel] Impossible to get stable?

Abdur-Rahman Advany rails at advany.com
Thu Sep 7 13:18:50 EDT 2006


Hey Eric,

Would you mind sharing this code with me? I am experiencing the same 
problems...

Abdul

Ezra Zygmuntowicz wrote:
> Hey Eric-
>
> 	Yeah I think your approach is valid. Would you mind sending me the  
> code you used for this experiment? I would like to run some tests  
> myself. I will add some unit tests to try and see what the upper  
> limits of threaded AR usage is.
>
> Thanks-
> -Ezra
>
> On Aug 20, 2006, at 5:20 AM, Erik Morton wrote:
>
>   
>> I'm planning on using Backgroundrb to run background processes for a
>> fairly large all-Rails application that is launching in two weeks.
>> I've been getting a bit paranoid about how ActiveRecord would react
>> with possibly as many as twenty workers chugging away at the same
>> time on the same tables. I put together a dead-simple "test" and the
>> results were, I think, very positive. (The OS is OSX 10.4.7)
>>
>> Setup: One worker class that queries a MySQL database (4.1.18) n
>> times and then calls ActiveRecord::Base.connection.disconnect! and
>> kill. The worker executes the "SHOW PROCESSLIST" query, which returns
>> a list of active MySQL processes, and logs the number of active
>> connections along with the to_s of the connection and the thread.
>>
>> The test script uses MiddleMan to instantiate 100 workers passing in
>> the worker's number as the number (10, 11, 12...110) of times the
>> worker should loop and query the database. I took this approach so
>> that the workers would be closing connections while other threads had
>> connections open.
>>
>> Goals: 1) verify that each Thread has his own connection. 2) verify
>> that each worker can call disconnect! to close their own connection
>> to free it up for other workers.
>>
>> Here's a snippet of the log file.
>>
>> worker number, ActiveRecord::Base.object_id,
>> ActiveRecord::Base.connection.to_s, Thread.current.to_s, # of active
>> MySQL connections
>> 25, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>> 0x277ba04>, #<Thread:0x277c4b8>, 12
>> 21, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>> 0x277a730>, #<Thread:0x277b0f4>, 12
>> 18, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>> 0x26c86e8>, #<Thread:0x26c8bfc>, 12
>> 17, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>> 0x26d61d0>, #<Thread:0x26d7620>, 12
>> 16, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>> 0x26e06d0>, #<Thread:0x26e0a18>, 12
>> 15, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>> 0x26e7098>, #<Thread:0x26e8218>, 12
>> 23, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>> 0x2784f8c>, #<Thread:0x27855a4>, 11
>>
>> In this rather unscientific way I have positively confirmed (I think)
>> that on Mac OSX 10.4.7, MySQL 4.1.18 and AR 1.14.4,  each worker gets
>> his own thread with his own connection, and can freely close his
>> connection without adversely affecting any other thread. Any thoughts
>> on this approach?
>>
>> I made one small change to swap out the SHOW PROCESS list query with
>> MyModel.find :first and the results were the same -- each worker had
>> his own thread with his own connection that he can close without
>> affecting any other threads.
>>
>> Finally, I altered the start script to set allow concurrency to
>> false, ran the test and not surprisingly all heck broke loose:
>>
>> 39, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>> 0x26fdca8>, #<Thread:0x2752aa0>, 1
>> 25, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>> 0x26fdca8>, #<Thread:0x2779a88>, 1
>> 86, 3705192,#<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>> 0x26fdca8>,#<Thread:0x2614cc4>,
>> 54, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>> 0x26fdca8>, #<Thread:0x271be9c>, 1
>> 87, 3705192,#<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>> 0x26fdca8>,#<Thread:0x26de9fc>,
>> Mysql::Error: MySQL server has gone away: SHOW PROCESSLIST; -
>> (ActiveRecord::StatementInvalid)
>> /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/lib/
>> active_record/connection_adapters/abstract_adapter.rb:120:in `log'
>> /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/lib/
>> active_record/connection_adapters/mysql_adapter.rb:184:in `execute'
>>
>> Different threads but the same connection and the first worker that
>> closes his connection ruins the whole party.
>>
>> Are results like the above consistent with the intention of
>> Backgroundrb? Is my approach was at least somewhat valid?
>>
>> Erik
>>
>> On Aug 20, 2006, at 12:39 AM, Ezra Zygmuntowicz wrote:
>>
>>     
>>> On Aug 18, 2006, at 6:20 PM, Ben Johnson wrote:
>>>
>>>       
>>>> I have concluded that it is impossible to get this background
>>>> process stable. I am out of ideas and could really use some help.
>>>>
>>>> Here is what I have:
>>>>
>>>> 50 workers, just running an infinite loop that constantly calls
>>>> save! on a model. Keep in mind this is just for testing purposes.
>>>> I catch all exceptions and put them in the log.
>>>>
>>>> I am tried setting allow_concurrency to true and false. Neither of
>>>> them is stable. For both true and false I have got the following
>>>> errors:
>>>>
>>>> #<ActiveRecord::StatementInvalid: Mysql::Error: Lock wait timeout
>>>> exceeded; try restarting transaction:
>>>>
>>>> #<ActiveRecord::StatementInvalid: Mysql::Error: MySQL server has
>>>> gone away:
>>>>
>>>> What have you found to be most stable? Setting allow concurrency
>>>> to false and just putting a Mutex.synchronize object around all
>>>> database calls? Or setting it true?
>>>>
>>>> Thank You,
>>>> Ben Johnson
>>>> E: bjohnson at contuitive.com
>>>> O: 800-341-6826
>>>> M: 817-229-4258
>>>>         
>>> Hey Ben-
>>>
>>> 	I feel your pain. I have come to the conclusion that ActiveRecord
>>> is just not thread safe no matter what you do. My best suggestion
>>> is to use either dbi or mysql bindings directly in your workers.
>>> These are thread safe and while you would have to write some sql
>>> yourself, it would solve your issue. I get the feeling that
>>> railscore is not very interested in making rails thread safe at
>>> this point and pushing as many db connections in as many threads as
>>> you are doing is more the ActiveRecord can handle.
>>>
>>> 	I know its not what you want to hear but its the situation we have
>>> to deal with.
>>>
>>> -Ezra
>>> _______________________________________________
>>> Backgroundrb-devel mailing list
>>> Backgroundrb-devel at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>>       
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>
>>     
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
>   



More information about the Backgroundrb-devel mailing list