[Backgroundrb-devel] adding results from threads to a collection and returning it

Stevie Clifton stevie at slowbicycle.com
Fri Jun 13 04:45:29 EDT 2008

Hey Hemant,

A couple of questions about fetch_parallely:

1)  Does it operate in the same way as thread_pool.defer, where the
number of concurrent threads are limited by :pool_size?

2)  Why did you choose to introduce another method instead of
providing a thread-safe register_status? (more out of curiosity than
anything else -- in my code I've overridden register_status to use a
Mutex, and am wondering what the benefit of fetch_parallely would be
to this)



On Thu, Jun 12, 2008 at 3:54 PM, hemant <gethemant at gmail.com> wrote:
> On Wed, Jun 11, 2008 at 5:26 AM, Neil Mock <neilmock at gmail.com> wrote:
>> Forgive me if this has been addressed somewhere, but I have searched and
>> can't come up with anything.
>> I am basically trying to distribute several web page scraping tasks among
>> different threads, and have the results from each added to an Array which is
>> ultimately returned by the backgroundrb worker.  Here is an example of what
>> I'm trying to do in a worker method:
>>      pages = Array.new
>>      pages_to_scrape.each do |url|
>>           thread_pool.defer(url) do |url|
>>             begin
>>               # model object performs the scraping
>>               page = ScrapedPage.new(page.url)
>>               pages << page
>>             rescue
>>               logger.info "page scrape failed"
>>             end
>>           end
>>         end
>>       end
>>     return pages
>> From monitoring the backgroundrb logs, it appears that all of the pages are
>> completed successfully in the threads.  However, the array that is returned
>> is empty.  This is to be expected I suppose because the threads don't
>> complete before the array is returned, but my question is: how can I make
>> the worker wait to return the array only when all of the threads are
>> complete?
> Neil,
> I have a solution for you in git version:
> http://gnufied.org/2008/06/12/unthreaded-threads-of-hobbiton/
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel

More information about the Backgroundrb-devel mailing list