[Backgroundrb-devel] Best way to run hundreds of concurrent tasks?

Robert Matei rmatei at gmail.com
Sun Sep 14 22:34:55 EDT 2008


Hi all,

I'm trying to figure out to run some asynchronous tasks in a Facebook app.
I've got things working, but BRB crashes after a little while, and I'm not
sure if my setup is ideal.

Here's the scenario:
- whenever a user visits my app, I need to fire off a bunch of API calls.
- these calls need to start right away, because the user sees a 'loading'
screen until they return.
- the job is not calculation or memory-intensive on my end, but it does take
15-20 seconds to get all the responses from Facebook.

I'm on a fairly small server, so it's important to keep memory use low (and
stability high), while still being able to handle thousands of simultaneous
visitors. Right now I'm using a single worker, and each visitor generates a
new thread in the thread pool. After a little while tasks only get a few
calls in before failing with a "Connection reset by peer" error. Eventually
my worker just disappears altogether.

What's going on? What kind of setup should I be running?

Thanks a lot for a great plugin...
Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080914/638c20a6/attachment.html>


More information about the Backgroundrb-devel mailing list