[Backgroundrb-devel] Using Backgroundrb?

Julien Genestoux julien.genestoux at gmail.com
Wed Apr 23 17:46:52 EDT 2008


Thanks guys... that's a ton of info! I am definetely gonna use the
thread_pool... as soon as I can find the documentation ;D

1- For each feed, I define a "frequency" (every minute, every hour...
every 30 minutes...) that will be updated every time I'm parsing the
feed: if the parser returns "new" element, I am increasding the
frequency (from 1 time per hour, to 1 time per 30 min.), if not, I'm
decreasing the frequency...

2- I also have a "last_update" field which remembers the time when the
feed was parsed for the last time.

3- With 1 & 2, I know how "late" I am to parse a feed... so when I
choose my next feed to parse, I am always choosing the one that is the
most "late"

I am not sure if Steevie's approach of having multiple tasks for the
worker applies here. Actually, I am not even schedulling my worker, I
am just launching it once, and the parse_feeds runs forever (while
true do... end)

Also, if I understand well Paul's code, his approach allows my worker
to be more efficient always, but doesn't take into account the
"lateness" of my feeds.


My idea would be to add/remove worker according to "how late" I am in
parsing feeds.
If my the the lastest feed is late by more than 10min, I would add one
worker... and If my latest feed is late by less than 5 minutes, I
would remove one worker

Does this approach makes sense to you?

Thanks a lot for your help guys...


On 4/23/08, Paul Kmiec <paul.kmiec at appfolio.com> wrote:
> You can use the built build thread pool to process more than one feed within
> the same worker. So within the worker, you'd do,
>
> def parse_feeds
>   loop do
>     feed = Feed.find_feed_to_process
>     thread_pool.defer do
>        feed.parse
>     end
>   end
> end
>
> I think the default pool size is 20. You can control the size of the thread
> pool using a class level method, as I recall it is
>
> pool_size x
>
> Paul
>
>
>  On Wed, Apr 23, 2008 at 7:30 AM, Julien Genestoux
> <julien.genestoux at gmail.com> wrote:
> > Thanks Adam,
> >
> > That sounded weird to me as well to have one worker for each feed...
> > However, if I only have one worker, that also means that I am parsing
> > one feed only at any moment. An option, is maybe to have a few workers
> > (denpending on the number of feeds)  that parse feeds concurrently?
> >
> > If I only have one worker, according to you what should be the
> > winnning strategy to choose the "right" parse to feed? Obviously some
> > feeds need to be parsed one every few minutes, while some other might
> > no need to be parse more than every hour...
> >
> > Any idea/tip on this?
> >
> >
> >
> >
> >
> >
> >
> > On 4/23/08, Adam Williams <adam at thewilliams.ws> wrote:
> > > On Apr 23, 2008, at 1:07 AM, Julien Genestoux wrote:
> > >
> > >  > I still have a few questions : shoud I have one worker for each feed
> > >  > that is called periodically (add_periodic_timer) or rather one single
> > >  > worker that calls every feed one by one?
> > >  >
> > >  > What is the best solution, perfomance-wise?
> > >
> > >
> > > Good question... I don't suppose I know exactly. I would start by
> > >  processing all the feeds in one worker invocation - that is what I
> > >  have done for sending an unknown amount of email. It just seems wrong
> > >  to me to invoke a worker for one email at a time.
> > >
> > >  The right answer likely lies in understanding the whole MasterWorker,
> > >  Packet::Reactor/handler_instance.ask_work bits of the
> puzzle...
> > >
> > >
> > >     adam
> > >
> > > _______________________________________________
> > >  Backgroundrb-devel mailing list
> > >  Backgroundrb-devel at rubyforge.org
> > >
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> > >
> >
> >
> >
> > --
> > --
> > Julien Genestoux
> > julien.genestoux at gmail.com
> > http://www.ouvre-boite.com
> > +1 (415) 254 7340
> > +33 (0)8 70 44 76 29
> > _______________________________________________
> >
> >
> >
> > Backgroundrb-devel mailing list
> > Backgroundrb-devel at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> >
>
>


-- 
--
Julien Genestoux
julien.genestoux at gmail.com
http://www.ouvre-boite.com
+1 (415) 254 7340
+33 (0)8 70 44 76 29


More information about the Backgroundrb-devel mailing list