[Backgroundrb-devel] MiddleMan.worker blocks?

[Marc Evans]

On Wed, 7 Mar 2007, Ezra Zygmuntowicz wrote:

>
> On Mar 7, 2007, at 1:46 PM, Mason Hale wrote:
>
>> 
>> On 3/7/07, Marc Evans <Marc at softwarehackery.com> wrote:
>> Notice in the above the "sleep 0.001". Without that sleep, I get the
>> blocking behavior I described in the original post. With it, I get the
>> behavior I would expect, which is that I can retrieve the progress
>> reasonably quickly and repeatedly.
>> 
>> Again, any suggestions?
>> 
>> That is very odd. My gut says it must have something to do with locking 
>> between threads.
>> I've previously run into thread-related issues around the worker results 
>> feature, finding
>> that it is unreliable. This smells like a similar problem.
>> 
>> more detail:
>> http://rubyforge.org/pipermail/backgroundrb-devel/2007-January/000638.html
>> http://backgroundrb.devjavu.com/projects/backgroundrb/ticket/43
>> 
>> My suggestion would be to store your worker state externally in a database 
>> or other store.
>> 
>> That "sleep 0.0001" works at all is weird, and I'd be hesitant to rely on 
>> it continuing to work.
>> 
>> Mason
>> 
>
> 	Actually the sleep makes sense. It would seem that your code is 
> blocking the thread which makes all the other threads block. Putting a sleep 
> in there gives the scheduler time to schedule the other threads to be run.
>
> Cheers-
>
> -- Ezra Zygmuntowicz-- Lead Rails Evangelist
> -- ez at engineyard.com
> -- Engine Yard, Serious Rails Hosting
> -- (866) 518-YARD (9273)
>

So, is there a "best practice" method of dealing with this? My worker code 
is in a tight loop, marshalling a few million records into XML, outputting 
the marshalled results to a file. I would have expected that each database 
query (I grab 5000 rows at a time) and each IO write would be a context 
switch candidate, but aparenetly not. Is there a call I should make to 
explicitely yield periodically?

- Marc