[Backgroundrb-devel] Impossible to get stable?

Ezra Zygmuntowicz ezmobius at gmail.com
Thu Sep 7 13:31:16 EDT 2006


Hey Abdur-

	Here is the code that he sent me when I requested it.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: backgroundrbtests.tar.gz
Type: application/x-gzip
Size: 1350 bytes
Desc: not available
Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060907/0e6e75bd/attachment-0001.gz 
-------------- next part --------------

-Ezra

On Sep 7, 2006, at 10:18 AM, Abdur-Rahman Advany wrote:

> Hey Eric,
>
> Would you mind sharing this code with me? I am experiencing the same
> problems...
>
> Abdul
>
> Ezra Zygmuntowicz wrote:
>> Hey Eric-
>>
>> 	Yeah I think your approach is valid. Would you mind sending me the
>> code you used for this experiment? I would like to run some tests
>> myself. I will add some unit tests to try and see what the upper
>> limits of threaded AR usage is.
>>
>> Thanks-
>> -Ezra
>>
>> On Aug 20, 2006, at 5:20 AM, Erik Morton wrote:
>>
>>
>>> I'm planning on using Backgroundrb to run background processes for a
>>> fairly large all-Rails application that is launching in two weeks.
>>> I've been getting a bit paranoid about how ActiveRecord would react
>>> with possibly as many as twenty workers chugging away at the same
>>> time on the same tables. I put together a dead-simple "test" and the
>>> results were, I think, very positive. (The OS is OSX 10.4.7)
>>>
>>> Setup: One worker class that queries a MySQL database (4.1.18) n
>>> times and then calls ActiveRecord::Base.connection.disconnect! and
>>> kill. The worker executes the "SHOW PROCESSLIST" query, which  
>>> returns
>>> a list of active MySQL processes, and logs the number of active
>>> connections along with the to_s of the connection and the thread.
>>>
>>> The test script uses MiddleMan to instantiate 100 workers passing in
>>> the worker's number as the number (10, 11, 12...110) of times the
>>> worker should loop and query the database. I took this approach so
>>> that the workers would be closing connections while other threads  
>>> had
>>> connections open.
>>>
>>> Goals: 1) verify that each Thread has his own connection. 2) verify
>>> that each worker can call disconnect! to close their own connection
>>> to free it up for other workers.
>>>
>>> Here's a snippet of the log file.
>>>
>>> worker number, ActiveRecord::Base.object_id,
>>> ActiveRecord::Base.connection.to_s, Thread.current.to_s, # of active
>>> MySQL connections
>>> 25, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>>> 0x277ba04>, #<Thread:0x277c4b8>, 12
>>> 21, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>>> 0x277a730>, #<Thread:0x277b0f4>, 12
>>> 18, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>>> 0x26c86e8>, #<Thread:0x26c8bfc>, 12
>>> 17, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>>> 0x26d61d0>, #<Thread:0x26d7620>, 12
>>> 16, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>>> 0x26e06d0>, #<Thread:0x26e0a18>, 12
>>> 15, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>>> 0x26e7098>, #<Thread:0x26e8218>, 12
>>> 23, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>>> 0x2784f8c>, #<Thread:0x27855a4>, 11
>>>
>>> In this rather unscientific way I have positively confirmed (I  
>>> think)
>>> that on Mac OSX 10.4.7, MySQL 4.1.18 and AR 1.14.4,  each worker  
>>> gets
>>> his own thread with his own connection, and can freely close his
>>> connection without adversely affecting any other thread. Any  
>>> thoughts
>>> on this approach?
>>>
>>> I made one small change to swap out the SHOW PROCESS list query with
>>> MyModel.find :first and the results were the same -- each worker had
>>> his own thread with his own connection that he can close without
>>> affecting any other threads.
>>>
>>> Finally, I altered the start script to set allow concurrency to
>>> false, ran the test and not surprisingly all heck broke loose:
>>>
>>> 39, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>>> 0x26fdca8>, #<Thread:0x2752aa0>, 1
>>> 25, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>>> 0x26fdca8>, #<Thread:0x2779a88>, 1
>>> 86, 3705192,#<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>>> 0x26fdca8>,#<Thread:0x2614cc4>,
>>> 54, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>>> 0x26fdca8>, #<Thread:0x271be9c>, 1
>>> 87, 3705192,#<ActiveRecord::ConnectionAdapters::MysqlAdapter:
>>> 0x26fdca8>,#<Thread:0x26de9fc>,
>>> Mysql::Error: MySQL server has gone away: SHOW PROCESSLIST; -
>>> (ActiveRecord::StatementInvalid)
>>> /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/lib/
>>> active_record/connection_adapters/abstract_adapter.rb:120:in `log'
>>> /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/lib/
>>> active_record/connection_adapters/mysql_adapter.rb:184:in `execute'
>>>
>>> Different threads but the same connection and the first worker that
>>> closes his connection ruins the whole party.
>>>
>>> Are results like the above consistent with the intention of
>>> Backgroundrb? Is my approach was at least somewhat valid?
>>>
>>> Erik
>>>
>>> On Aug 20, 2006, at 12:39 AM, Ezra Zygmuntowicz wrote:
>>>
>>>
>>>> On Aug 18, 2006, at 6:20 PM, Ben Johnson wrote:
>>>>
>>>>
>>>>> I have concluded that it is impossible to get this background
>>>>> process stable. I am out of ideas and could really use some help.
>>>>>
>>>>> Here is what I have:
>>>>>
>>>>> 50 workers, just running an infinite loop that constantly calls
>>>>> save! on a model. Keep in mind this is just for testing purposes.
>>>>> I catch all exceptions and put them in the log.
>>>>>
>>>>> I am tried setting allow_concurrency to true and false. Neither of
>>>>> them is stable. For both true and false I have got the following
>>>>> errors:
>>>>>
>>>>> #<ActiveRecord::StatementInvalid: Mysql::Error: Lock wait timeout
>>>>> exceeded; try restarting transaction:
>>>>>
>>>>> #<ActiveRecord::StatementInvalid: Mysql::Error: MySQL server has
>>>>> gone away:
>>>>>
>>>>> What have you found to be most stable? Setting allow concurrency
>>>>> to false and just putting a Mutex.synchronize object around all
>>>>> database calls? Or setting it true?
>>>>>
>>>>> Thank You,
>>>>> Ben Johnson
>>>>> E: bjohnson at contuitive.com
>>>>> O: 800-341-6826
>>>>> M: 817-229-4258
>>>>>
>>>> Hey Ben-
>>>>
>>>> 	I feel your pain. I have come to the conclusion that ActiveRecord
>>>> is just not thread safe no matter what you do. My best suggestion
>>>> is to use either dbi or mysql bindings directly in your workers.
>>>> These are thread safe and while you would have to write some sql
>>>> yourself, it would solve your issue. I get the feeling that
>>>> railscore is not very interested in making rails thread safe at
>>>> this point and pushing as many db connections in as many threads as
>>>> you are doing is more the ActiveRecord can handle.
>>>>
>>>> 	I know its not what you want to hear but its the situation we have
>>>> to deal with.
>>>>
>>>> -Ezra
>>>> _______________________________________________
>>>> Backgroundrb-devel mailing list
>>>> Backgroundrb-devel at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>>>
>>> _______________________________________________
>>> Backgroundrb-devel mailing list
>>> Backgroundrb-devel at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>>
>>>
>>
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>
>>
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel



More information about the Backgroundrb-devel mailing list