[Backgroundrb-devel] Question about backgroundrb and resource usage

Peter Gengler ppgengler at prevailhs.com
Mon Aug 10 20:17:18 EDT 2009


You know that's an interesting point about the single worker; has anyone ever used Skynet?  Their fortay is distributing map/reduce problems in the background, but they also mixed in a function to ActiveRecord called send_later which allowed you to send off any method of a model for later processing.  I wrote my own mixin that does the same thing with Backgroundrb and a GenericWorker class (and I added send_now_or_later so I could have a fallback) because I liked keeping the functions on the model.  That would lend itself to reducing the backgroundrb overhead as you'd have a single worker for any asynchronous AR processing, although it doesn't help with the cron-style jobs.

\Peter

-----Original Message-----
From: Adam Williams [mailto:adam at thewilliams.ws] 
Sent: Monday, August 10, 2009 6:50 PM
To: Peter Gengler
Cc: backgroundrb-devel at rubyforge.org
Subject: Re: [Backgroundrb-devel] Question about backgroundrb and resource usage

On Aug 10, 2009, at 3:44 PM, Peter Gengler wrote:

> So I have a somewhat general question about backgroundrb and  
> resource usage.  I hadn't really considered the fact that it spools  
> up a new rails process for each worker until I started making more  
> workers; now that I'm have half a dozen it seems like the current  
> design is really unscalable.  I.e. if my rails stack takes 200MB of  
> memory for example, that means 10 workers use 2GB of memory?  That  
> seems overly excessive to me, am I missing something that prevents  
> that resource bloat?

As far as I have seen, you're not missing anything. I had the same  
problem and ended up having a single worker, with different methods  
for the work. I'm considering getting rid of backgroundrb altogether,  
favoring instead simply loading ActiveRecord and the model classes I  
need, with the hope that a cron task can fire the thing up within a  
few seconds. We need tasks to run every minute, so it must be fast  
loading. Anyway, that would allow us to unload a couple hundred meg at  
Engine Yard.

   adam


More information about the Backgroundrb-devel mailing list