[Backgroundrb-devel] Thread Pool Size?

[Brian Mulloy]

>         You are using the latest release of backgroundrb?


module BackgrounDRb
  VERSION = "0.2.0"
end

doh, I am going to upgrade and then give it another go.

these are excellent thoughts, Ezra, thanks.


0.2.1? The pool
> size works like this.. It is supposed to only allow the number you
> specify of workers to be running at one time.



Any more requests for
> new workers when the pool is full get queued up and then they are run
> when another worker leaves the pool and dies.



What exactly is
> happening to you? Are you sending a bunch of requests and they all
> run at once even though you have set the limit?
>
>         It may be that we need to add some tighter control to how many
> active jobs there are. Maybe a way for new_worker to return an error
> that you can rescue and retry on a different drb server? The current
> thread pool should limit the actively running workers but writing
> properly threaded code is hard so I wouldn't be too surprised if I
> made a mistake.
>
>         I can imagine a way to be more strict about the pool limit but it
> will take some doing I imagine. Can you explain how your bdrb stuff
> works? Are you starting a new worker for each time you do something?
> Or do you have a certain number of workers that always run and keep
> running in a loop accepting jobs? The reason I ask is that the way
> that the new backgroundrb works its better to have a number of named
> workers that always run in a loop accepting jobs from a queue. So
> instead of calling new_worker all the time you set your workers to
> autostart at server start time and then just call methods on the
> daemon style workers. This way you can start exactly how many workers
> you want and give each one a named job key. Then you can either have
> the workers running in a loop where they wake up every second or
> whatever interval you set and do whatever jobs they need to do,
>
>         Since you say that you are already keeping the state of the
> workers
> in the database then maybe an approach like this would be better
> suited. The basic idea is something like this:
>
> An example worker that pulls events off the db and executes them,
> then saves them out and loops again
>
> class RSSWorker < BackgrounDRb::Worker::RailsBase
>    require 'rss'
>    require 'net/http'
>
>    def do_work(args)
>      @args = args
>      mainloop
>    end
>
>    def mainloop
>      loop {
>        sleep @args[:sleep]
>        RssUrls.find_all_by_pending(true).each do |rss|
>          rss.processing=true
>          rss.save
>          Net::HTTP.start(rss.host) do |http|
>            response = http.get(rss.path)
>            raise response.code unless response.code == "200"
>            rss_parser = RSS::Parser.new(response.body)
>            rss.output = rss_parser.parse
>          end
>          rss.completed=true
>          rss.save
>        end
>      }
>    end
>
> end
> RSSWorker.register
>
>         Something like that may work better although I am not certain
> exactly what your workers need to do. You coudl also do the same
> thing with long lived workers but create a method other then do_work
> which you can call from rails via the work_thread method.
>
>
> Cheers-
> -- Ezra Zygmuntowicz
> -- Lead Rails Evangelist
> -- ez at engineyard.com
> -- Engine Yard, Serious Rails Hosting
> -- (866) 518-YARD (9273)
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061208/9d575aab/attachment.html