[Backgroundrb-devel] MiddleMan.worker blocks?

Ezra Zygmuntowicz ezmobius at gmail.com
Wed Mar 7 17:13:40 EST 2007

On Mar 7, 2007, at 2:07 PM, Marc Evans wrote:

> On Wed, 7 Mar 2007, Ezra Zygmuntowicz wrote:
>> On Mar 7, 2007, at 1:46 PM, Mason Hale wrote:
>>> On 3/7/07, Marc Evans <Marc at softwarehackery.com> wrote:
>>> Notice in the above the "sleep 0.001". Without that sleep, I get the
>>> blocking behavior I described in the original post. With it, I  
>>> get the
>>> behavior I would expect, which is that I can retrieve the progress
>>> reasonably quickly and repeatedly.
>>> Again, any suggestions?
>>> That is very odd. My gut says it must have something to do with  
>>> locking between threads.
>>> I've previously run into thread-related issues around the worker  
>>> results feature, finding
>>> that it is unreliable. This smells like a similar problem.
>>> more detail:
>>> http://rubyforge.org/pipermail/backgroundrb-devel/2007-January/ 
>>> 000638.html
>>> http://backgroundrb.devjavu.com/projects/backgroundrb/ticket/43
>>> My suggestion would be to store your worker state externally in a  
>>> database or other store.
>>> That "sleep 0.0001" works at all is weird, and I'd be hesitant to  
>>> rely on it continuing to work.
>>> Mason
>> 	Actually the sleep makes sense. It would seem that your code is  
>> blocking the thread which makes all the other threads block.  
>> Putting a sleep in there gives the scheduler time to schedule the  
>> other threads to be run.
>> Cheers-
>> -- Ezra Zygmuntowicz-- Lead Rails Evangelist
>> -- ez at engineyard.com
>> -- Engine Yard, Serious Rails Hosting
>> -- (866) 518-YARD (9273)
> So, is there a "best practice" method of dealing with this? My  
> worker code is in a tight loop, marshalling a few million records  
> into XML, outputting the marshalled results to a file. I would have  
> expected that each database query (I grab 5000 rows at a time) and  
> each IO write would be a context switch candidate, but aparenetly  
> not. Is there a call I should make to explicitely yield periodically?
> - Marc


	It's hard to say anything about a best practice here. I have never  
had to do a sleep0 to get other threads scheduled but I am carefull  
to not use anything that is blocking IO. Ruby's green threads are  
admittedly weak sauce :( If you do a blocking IO call in one thread  
*all* threads will block and none will schedule until it finishes.

	When you say marshalling millions of records to xml I think you may  
just bee swamping the hell out of the interpreter and its blocking in  
that thread for some reason. I think in your case putting the sleep 0  
in there to get other threads scheduled is a hack at best but I think  
it's the best hack in this case right now.

-- Ezra Zygmuntowicz 
-- Lead Rails Evangelist
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)

More information about the Backgroundrb-devel mailing list