[Backgroundrb-devel] MiddleMan.worker blocks?
ezmobius at gmail.com
Wed Mar 7 17:13:40 EST 2007
On Mar 7, 2007, at 2:07 PM, Marc Evans wrote:
> On Wed, 7 Mar 2007, Ezra Zygmuntowicz wrote:
>> On Mar 7, 2007, at 1:46 PM, Mason Hale wrote:
>>> On 3/7/07, Marc Evans <Marc at softwarehackery.com> wrote:
>>> Notice in the above the "sleep 0.001". Without that sleep, I get the
>>> blocking behavior I described in the original post. With it, I
>>> get the
>>> behavior I would expect, which is that I can retrieve the progress
>>> reasonably quickly and repeatedly.
>>> Again, any suggestions?
>>> That is very odd. My gut says it must have something to do with
>>> locking between threads.
>>> I've previously run into thread-related issues around the worker
>>> results feature, finding
>>> that it is unreliable. This smells like a similar problem.
>>> more detail:
>>> My suggestion would be to store your worker state externally in a
>>> database or other store.
>>> That "sleep 0.0001" works at all is weird, and I'd be hesitant to
>>> rely on it continuing to work.
>> Actually the sleep makes sense. It would seem that your code is
>> blocking the thread which makes all the other threads block.
>> Putting a sleep in there gives the scheduler time to schedule the
>> other threads to be run.
>> -- Ezra Zygmuntowicz-- Lead Rails Evangelist
>> -- ez at engineyard.com
>> -- Engine Yard, Serious Rails Hosting
>> -- (866) 518-YARD (9273)
> So, is there a "best practice" method of dealing with this? My
> worker code is in a tight loop, marshalling a few million records
> into XML, outputting the marshalled results to a file. I would have
> expected that each database query (I grab 5000 rows at a time) and
> each IO write would be a context switch candidate, but aparenetly
> not. Is there a call I should make to explicitely yield periodically?
> - Marc
It's hard to say anything about a best practice here. I have never
had to do a sleep0 to get other threads scheduled but I am carefull
to not use anything that is blocking IO. Ruby's green threads are
admittedly weak sauce :( If you do a blocking IO call in one thread
*all* threads will block and none will schedule until it finishes.
When you say marshalling millions of records to xml I think you may
just bee swamping the hell out of the interpreter and its blocking in
that thread for some reason. I think in your case putting the sleep 0
in there to get other threads scheduled is a hack at best but I think
it's the best hack in this case right now.
-- Ezra Zygmuntowicz
-- Lead Rails Evangelist
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)
More information about the Backgroundrb-devel