[Backgroundrb-devel] MiddleMan.worker blocks?
Marc at SoftwareHackery.Com
Wed Mar 7 17:07:52 EST 2007
On Wed, 7 Mar 2007, Ezra Zygmuntowicz wrote:
> On Mar 7, 2007, at 1:46 PM, Mason Hale wrote:
>> On 3/7/07, Marc Evans <Marc at softwarehackery.com> wrote:
>> Notice in the above the "sleep 0.001". Without that sleep, I get the
>> blocking behavior I described in the original post. With it, I get the
>> behavior I would expect, which is that I can retrieve the progress
>> reasonably quickly and repeatedly.
>> Again, any suggestions?
>> That is very odd. My gut says it must have something to do with locking
>> between threads.
>> I've previously run into thread-related issues around the worker results
>> feature, finding
>> that it is unreliable. This smells like a similar problem.
>> more detail:
>> My suggestion would be to store your worker state externally in a database
>> or other store.
>> That "sleep 0.0001" works at all is weird, and I'd be hesitant to rely on
>> it continuing to work.
> Actually the sleep makes sense. It would seem that your code is
> blocking the thread which makes all the other threads block. Putting a sleep
> in there gives the scheduler time to schedule the other threads to be run.
> -- Ezra Zygmuntowicz-- Lead Rails Evangelist
> -- ez at engineyard.com
> -- Engine Yard, Serious Rails Hosting
> -- (866) 518-YARD (9273)
So, is there a "best practice" method of dealing with this? My worker code
is in a tight loop, marshalling a few million records into XML, outputting
the marshalled results to a file. I would have expected that each database
query (I grab 5000 rows at a time) and each IO write would be a context
switch candidate, but aparenetly not. Is there a call I should make to
explicitely yield periodically?
More information about the Backgroundrb-devel