[Backgroundrb-devel] Upper limit to number of jobs queued by BDRb?
gethemant at gmail.com
Fri May 30 12:17:20 EDT 2008
On Fri, May 30, 2008 at 9:35 AM, Raghu Srinivasan
<raghu.srinivasan at gmail.com> wrote:
> I use BDRb to process RSS feeds for users on my site (http://feedflix.com).
> I have a batch job that queries the DB for records that haven't been updated
> in the last so many hours and kicks off a background job for each of them.
> If N records are returned by the DB, N background jobs gets queued and get
> done serially. As long as N is 255 or under, everything works like a charm.
> I've noticed that whenever N is >= 256 (2 power 8), then at the 257th job
> BDRb stops processing any more users. I can get around it by limiting the DB
> query to return no more than 255 records and then all is fine. No problems
> at all. But over that, I see this issue. Repeatedly.
How are you queuing the jobs? Are you using thread_pool? I am afraid,
it could be because of restriction in number of open file descriptors
open to 1024.
More information about the Backgroundrb-devel