[Backgroundrb-devel] Best way to run hundreds of concurrent tasks?
gethemant at gmail.com
Mon Sep 15 00:30:29 EDT 2008
On Sun, 2008-09-14 at 19:34 -0700, Robert Matei wrote:
> Hi all,
> I'm trying to figure out to run some asynchronous tasks in a Facebook
> app. I've got things working, but BRB crashes after a little while,
> and I'm not sure if my setup is ideal.
> Here's the scenario:
> - whenever a user visits my app, I need to fire off a bunch of API
> - these calls need to start right away, because the user sees a
> 'loading' screen until they return.
> - the job is not calculation or memory-intensive on my end, but it
> does take 15-20 seconds to get all the responses from Facebook.
> I'm on a fairly small server, so it's important to keep memory use low
> (and stability high), while still being able to handle thousands of
> simultaneous visitors. Right now I'm using a single worker, and each
> visitor generates a new thread in the thread pool. After a little
> while tasks only get a few calls in before failing with a "Connection
> reset by peer" error. Eventually my worker just disappears
> What's going on? What kind of setup should I be running?
What version of BDRB? What version of packet gem? Which OS?
Are you using mechanize or something for interacting with facebook? or
just normal RFacebook or something?
More information about the Backgroundrb-devel