[Backgroundrb-devel] Using Backgroundrb?
stevie at animoto.com
Wed Apr 23 12:14:24 EDT 2008
There was a great thread about a similar situation about 10 days ago.
Check it out here:
Julien, you definitely don't want a worker for each feed, and you'll
want to use thread_pool.defer, which will allow you to concurrently
process as many feeds as you want (or as many as your system can
handle). From what you've said, it sounds like you'll only need one
worker coded up, but probably set multiple periodic timers (e.g one of
hourly parsing of high-priority feeds, one for nightlies, etc). The
method you specify in the periodic timer should use thread_pool.defer
to handle processing of multiple feeds at a time -- there's no reason
to do them sequentially.
On Wed, Apr 23, 2008 at 10:30 AM, Julien Genestoux
<julien.genestoux at gmail.com> wrote:
> Thanks Adam,
> That sounded weird to me as well to have one worker for each feed...
> However, if I only have one worker, that also means that I am parsing
> one feed only at any moment. An option, is maybe to have a few workers
> (denpending on the number of feeds) that parse feeds concurrently?
> If I only have one worker, according to you what should be the
> winnning strategy to choose the "right" parse to feed? Obviously some
> feeds need to be parsed one every few minutes, while some other might
> no need to be parse more than every hour...
> Any idea/tip on this?
> On 4/23/08, Adam Williams <adam at thewilliams.ws> wrote:
> > On Apr 23, 2008, at 1:07 AM, Julien Genestoux wrote:
> > > I still have a few questions : shoud I have one worker for each feed
> > > that is called periodically (add_periodic_timer) or rather one single
> > > worker that calls every feed one by one?
> > >
> > > What is the best solution, perfomance-wise?
> > Good question... I don't suppose I know exactly. I would start by
> > processing all the feeds in one worker invocation - that is what I
> > have done for sending an unknown amount of email. It just seems wrong
> > to me to invoke a worker for one email at a time.
> > The right answer likely lies in understanding the whole MasterWorker,
> > Packet::Reactor/handler_instance.ask_work bits of the puzzle...
> > adam
> > _______________________________________________
> > Backgroundrb-devel mailing list
> > Backgroundrb-devel at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> Julien Genestoux
> julien.genestoux at gmail.com
> +1 (415) 254 7340
> +33 (0)8 70 44 76 29
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
More information about the Backgroundrb-devel