[Backgroundrb-devel] Impossible to get stable?

Ezra Zygmuntowicz ezmobius at gmail.com
Sun Aug 20 14:41:49 EDT 2006


Hey Eric-

	Yeah I think your approach is valid. Would you mind sending me the  
code you used for this experiment? I would like to run some tests  
myself. I will add some unit tests to try and see what the upper  
limits of threaded AR usage is.

Thanks-
-Ezra

On Aug 20, 2006, at 5:20 AM, Erik Morton wrote:

> I'm planning on using Backgroundrb to run background processes for a
> fairly large all-Rails application that is launching in two weeks.
> I've been getting a bit paranoid about how ActiveRecord would react
> with possibly as many as twenty workers chugging away at the same
> time on the same tables. I put together a dead-simple "test" and the
> results were, I think, very positive. (The OS is OSX 10.4.7)
>
> Setup: One worker class that queries a MySQL database (4.1.18) n
> times and then calls ActiveRecord::Base.connection.disconnect! and
> kill. The worker executes the "SHOW PROCESSLIST" query, which returns
> a list of active MySQL processes, and logs the number of active
> connections along with the to_s of the connection and the thread.
>
> The test script uses MiddleMan to instantiate 100 workers passing in
> the worker's number as the number (10, 11, 12...110) of times the
> worker should loop and query the database. I took this approach so
> that the workers would be closing connections while other threads had
> connections open.
>
> Goals: 1) verify that each Thread has his own connection. 2) verify
> that each worker can call disconnect! to close their own connection
> to free it up for other workers.
>
> Here's a snippet of the log file.
>
> worker number, ActiveRecord::Base.object_id,
> ActiveRecord::Base.connection.to_s, Thread.current.to_s, # of active
> MySQL connections
> 25, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
> 0x277ba04>, #<Thread:0x277c4b8>, 12
> 21, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
> 0x277a730>, #<Thread:0x277b0f4>, 12
> 18, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
> 0x26c86e8>, #<Thread:0x26c8bfc>, 12
> 17, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
> 0x26d61d0>, #<Thread:0x26d7620>, 12
> 16, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
> 0x26e06d0>, #<Thread:0x26e0a18>, 12
> 15, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
> 0x26e7098>, #<Thread:0x26e8218>, 12
> 23, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
> 0x2784f8c>, #<Thread:0x27855a4>, 11
>
> In this rather unscientific way I have positively confirmed (I think)
> that on Mac OSX 10.4.7, MySQL 4.1.18 and AR 1.14.4,  each worker gets
> his own thread with his own connection, and can freely close his
> connection without adversely affecting any other thread. Any thoughts
> on this approach?
>
> I made one small change to swap out the SHOW PROCESS list query with
> MyModel.find :first and the results were the same -- each worker had
> his own thread with his own connection that he can close without
> affecting any other threads.
>
> Finally, I altered the start script to set allow concurrency to
> false, ran the test and not surprisingly all heck broke loose:
>
> 39, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
> 0x26fdca8>, #<Thread:0x2752aa0>, 1
> 25, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
> 0x26fdca8>, #<Thread:0x2779a88>, 1
> 86, 3705192,#<ActiveRecord::ConnectionAdapters::MysqlAdapter:
> 0x26fdca8>,#<Thread:0x2614cc4>,
> 54, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
> 0x26fdca8>, #<Thread:0x271be9c>, 1
> 87, 3705192,#<ActiveRecord::ConnectionAdapters::MysqlAdapter:
> 0x26fdca8>,#<Thread:0x26de9fc>,
> Mysql::Error: MySQL server has gone away: SHOW PROCESSLIST; -
> (ActiveRecord::StatementInvalid)
> /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/lib/
> active_record/connection_adapters/abstract_adapter.rb:120:in `log'
> /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/lib/
> active_record/connection_adapters/mysql_adapter.rb:184:in `execute'
>
> Different threads but the same connection and the first worker that
> closes his connection ruins the whole party.
>
> Are results like the above consistent with the intention of
> Backgroundrb? Is my approach was at least somewhat valid?
>
> Erik
>
> On Aug 20, 2006, at 12:39 AM, Ezra Zygmuntowicz wrote:
>
>>
>> On Aug 18, 2006, at 6:20 PM, Ben Johnson wrote:
>>
>>> I have concluded that it is impossible to get this background
>>> process stable. I am out of ideas and could really use some help.
>>>
>>> Here is what I have:
>>>
>>> 50 workers, just running an infinite loop that constantly calls
>>> save! on a model. Keep in mind this is just for testing purposes.
>>> I catch all exceptions and put them in the log.
>>>
>>> I am tried setting allow_concurrency to true and false. Neither of
>>> them is stable. For both true and false I have got the following
>>> errors:
>>>
>>> #<ActiveRecord::StatementInvalid: Mysql::Error: Lock wait timeout
>>> exceeded; try restarting transaction:
>>>
>>> #<ActiveRecord::StatementInvalid: Mysql::Error: MySQL server has
>>> gone away:
>>>
>>> What have you found to be most stable? Setting allow concurrency
>>> to false and just putting a Mutex.synchronize object around all
>>> database calls? Or setting it true?
>>>
>>> Thank You,
>>> Ben Johnson
>>> E: bjohnson at contuitive.com
>>> O: 800-341-6826
>>> M: 817-229-4258
>>
>>
>> Hey Ben-
>>
>> 	I feel your pain. I have come to the conclusion that ActiveRecord
>> is just not thread safe no matter what you do. My best suggestion
>> is to use either dbi or mysql bindings directly in your workers.
>> These are thread safe and while you would have to write some sql
>> yourself, it would solve your issue. I get the feeling that
>> railscore is not very interested in making rails thread safe at
>> this point and pushing as many db connections in as many threads as
>> you are doing is more the ActiveRecord can handle.
>>
>> 	I know its not what you want to hear but its the situation we have
>> to deal with.
>>
>> -Ezra
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>



More information about the Backgroundrb-devel mailing list