[Backgroundrb-devel] Impossible to get stable?

Erik Morton eimorton at gmail.com
Sun Aug 20 08:20:45 EDT 2006

I'm planning on using Backgroundrb to run background processes for a  
fairly large all-Rails application that is launching in two weeks.  
I've been getting a bit paranoid about how ActiveRecord would react  
with possibly as many as twenty workers chugging away at the same  
time on the same tables. I put together a dead-simple "test" and the  
results were, I think, very positive. (The OS is OSX 10.4.7)

Setup: One worker class that queries a MySQL database (4.1.18) n  
times and then calls ActiveRecord::Base.connection.disconnect! and  
kill. The worker executes the "SHOW PROCESSLIST" query, which returns  
a list of active MySQL processes, and logs the number of active  
connections along with the to_s of the connection and the thread.

The test script uses MiddleMan to instantiate 100 workers passing in  
the worker's number as the number (10, 11, 12...110) of times the  
worker should loop and query the database. I took this approach so  
that the workers would be closing connections while other threads had  
connections open.

Goals: 1) verify that each Thread has his own connection. 2) verify  
that each worker can call disconnect! to close their own connection  
to free it up for other workers.

Here's a snippet of the log file.

worker number, ActiveRecord::Base.object_id,  
ActiveRecord::Base.connection.to_s, Thread.current.to_s, # of active  
MySQL connections
25, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter: 
0x277ba04>, #<Thread:0x277c4b8>, 12
21, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter: 
0x277a730>, #<Thread:0x277b0f4>, 12
18, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter: 
0x26c86e8>, #<Thread:0x26c8bfc>, 12
17, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter: 
0x26d61d0>, #<Thread:0x26d7620>, 12
16, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter: 
0x26e06d0>, #<Thread:0x26e0a18>, 12
15, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter: 
0x26e7098>, #<Thread:0x26e8218>, 12
23, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter: 
0x2784f8c>, #<Thread:0x27855a4>, 11

In this rather unscientific way I have positively confirmed (I think)  
that on Mac OSX 10.4.7, MySQL 4.1.18 and AR 1.14.4,  each worker gets  
his own thread with his own connection, and can freely close his  
connection without adversely affecting any other thread. Any thoughts  
on this approach?

I made one small change to swap out the SHOW PROCESS list query with  
MyModel.find :first and the results were the same -- each worker had  
his own thread with his own connection that he can close without  
affecting any other threads.

Finally, I altered the start script to set allow concurrency to  
false, ran the test and not surprisingly all heck broke loose:

39, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter: 
0x26fdca8>, #<Thread:0x2752aa0>, 1
25, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter: 
0x26fdca8>, #<Thread:0x2779a88>, 1
86, 3705192,#<ActiveRecord::ConnectionAdapters::MysqlAdapter: 
54, 3705192, #<ActiveRecord::ConnectionAdapters::MysqlAdapter: 
0x26fdca8>, #<Thread:0x271be9c>, 1
87, 3705192,#<ActiveRecord::ConnectionAdapters::MysqlAdapter: 
Mysql::Error: MySQL server has gone away: SHOW PROCESSLIST; -  
active_record/connection_adapters/abstract_adapter.rb:120:in `log'
active_record/connection_adapters/mysql_adapter.rb:184:in `execute'

Different threads but the same connection and the first worker that  
closes his connection ruins the whole party.

Are results like the above consistent with the intention of  
Backgroundrb? Is my approach was at least somewhat valid?


On Aug 20, 2006, at 12:39 AM, Ezra Zygmuntowicz wrote:

> On Aug 18, 2006, at 6:20 PM, Ben Johnson wrote:
>> I have concluded that it is impossible to get this background  
>> process stable. I am out of ideas and could really use some help.
>> Here is what I have:
>> 50 workers, just running an infinite loop that constantly calls  
>> save! on a model. Keep in mind this is just for testing purposes.  
>> I catch all exceptions and put them in the log.
>> I am tried setting allow_concurrency to true and false. Neither of  
>> them is stable. For both true and false I have got the following  
>> errors:
>> #<ActiveRecord::StatementInvalid: Mysql::Error: Lock wait timeout  
>> exceeded; try restarting transaction:
>> #<ActiveRecord::StatementInvalid: Mysql::Error: MySQL server has  
>> gone away:
>> What have you found to be most stable? Setting allow concurrency  
>> to false and just putting a Mutex.synchronize object around all  
>> database calls? Or setting it true?
>> Thank You,
>> Ben Johnson
>> E: bjohnson at contuitive.com
>> O: 800-341-6826
>> M: 817-229-4258
> Hey Ben-
> 	I feel your pain. I have come to the conclusion that ActiveRecord  
> is just not thread safe no matter what you do. My best suggestion  
> is to use either dbi or mysql bindings directly in your workers.  
> These are thread safe and while you would have to write some sql  
> yourself, it would solve your issue. I get the feeling that  
> railscore is not very interested in making rails thread safe at  
> this point and pushing as many db connections in as many threads as  
> you are doing is more the ActiveRecord can handle.
> 	I know its not what you want to hear but its the situation we have  
> to deal with.
> -Ezra
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel

More information about the Backgroundrb-devel mailing list