[Backgroundrb-devel] Best way to run hundreds of concurrent tasks?
rmatei at gmail.com
Sun Sep 14 22:34:55 EDT 2008
I'm trying to figure out to run some asynchronous tasks in a Facebook app.
I've got things working, but BRB crashes after a little while, and I'm not
sure if my setup is ideal.
Here's the scenario:
- whenever a user visits my app, I need to fire off a bunch of API calls.
- these calls need to start right away, because the user sees a 'loading'
screen until they return.
- the job is not calculation or memory-intensive on my end, but it does take
15-20 seconds to get all the responses from Facebook.
I'm on a fairly small server, so it's important to keep memory use low (and
stability high), while still being able to handle thousands of simultaneous
visitors. Right now I'm using a single worker, and each visitor generates a
new thread in the thread pool. After a little while tasks only get a few
calls in before failing with a "Connection reset by peer" error. Eventually
my worker just disappears altogether.
What's going on? What kind of setup should I be running?
Thanks a lot for a great plugin...
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Backgroundrb-devel