From adam at thewilliams.ws Mon Aug 10 19:49:40 2009 From: adam at thewilliams.ws (Adam Williams) Date: Mon, 10 Aug 2009 18:49:40 -0500 Subject: [Backgroundrb-devel] Question about backgroundrb and resource usage In-Reply-To: References: Message-ID: <80814BA8-6466-4A82-8E52-A3F6FCDCA239@thewilliams.ws> On Aug 10, 2009, at 3:44 PM, Peter Gengler wrote: > So I have a somewhat general question about backgroundrb and > resource usage. I hadn?t really considered the fact that it spools > up a new rails process for each worker until I started making more > workers; now that I?m have half a dozen it seems like the current > design is really unscalable. I.e. if my rails stack takes 200MB of > memory for example, that means 10 workers use 2GB of memory? That > seems overly excessive to me, am I missing something that prevents > that resource bloat? As far as I have seen, you're not missing anything. I had the same problem and ended up having a single worker, with different methods for the work. I'm considering getting rid of backgroundrb altogether, favoring instead simply loading ActiveRecord and the model classes I need, with the hope that a cron task can fire the thing up within a few seconds. We need tasks to run every minute, so it must be fast loading. Anyway, that would allow us to unload a couple hundred meg at Engine Yard. adam From ppgengler at prevailhs.com Mon Aug 10 20:17:18 2009 From: ppgengler at prevailhs.com (Peter Gengler) Date: Mon, 10 Aug 2009 17:17:18 -0700 Subject: [Backgroundrb-devel] Question about backgroundrb and resource usage In-Reply-To: <80814BA8-6466-4A82-8E52-A3F6FCDCA239@thewilliams.ws> References: <80814BA8-6466-4A82-8E52-A3F6FCDCA239@thewilliams.ws> Message-ID: You know that's an interesting point about the single worker; has anyone ever used Skynet? Their fortay is distributing map/reduce problems in the background, but they also mixed in a function to ActiveRecord called send_later which allowed you to send off any method of a model for later processing. I wrote my own mixin that does the same thing with Backgroundrb and a GenericWorker class (and I added send_now_or_later so I could have a fallback) because I liked keeping the functions on the model. That would lend itself to reducing the backgroundrb overhead as you'd have a single worker for any asynchronous AR processing, although it doesn't help with the cron-style jobs. \Peter -----Original Message----- From: Adam Williams [mailto:adam at thewilliams.ws] Sent: Monday, August 10, 2009 6:50 PM To: Peter Gengler Cc: backgroundrb-devel at rubyforge.org Subject: Re: [Backgroundrb-devel] Question about backgroundrb and resource usage On Aug 10, 2009, at 3:44 PM, Peter Gengler wrote: > So I have a somewhat general question about backgroundrb and > resource usage. I hadn't really considered the fact that it spools > up a new rails process for each worker until I started making more > workers; now that I'm have half a dozen it seems like the current > design is really unscalable. I.e. if my rails stack takes 200MB of > memory for example, that means 10 workers use 2GB of memory? That > seems overly excessive to me, am I missing something that prevents > that resource bloat? As far as I have seen, you're not missing anything. I had the same problem and ended up having a single worker, with different methods for the work. I'm considering getting rid of backgroundrb altogether, favoring instead simply loading ActiveRecord and the model classes I need, with the hope that a cron task can fire the thing up within a few seconds. We need tasks to run every minute, so it must be fast loading. Anyway, that would allow us to unload a couple hundred meg at Engine Yard. adam From vaughan.webfu at gmail.com Mon Aug 10 22:16:42 2009 From: vaughan.webfu at gmail.com (Vaughan Magnusson) Date: Tue, 11 Aug 2009 14:16:42 +1200 Subject: [Backgroundrb-devel] monit paths Message-ID: Hi everyone, I'm trying to use monit to monitor a backgroundrb process (that gets killed occasionally), but I cant seem to seem the start or stop commands to work (i simply get 'execution failed') - though they do work fine from the command prompt. I'm not doing anything particularly sophisticated: check process backgroundrb_11006 with pidfile "/home/user/site/tmp/pids/backgroundrb_11006.pid" start = "/usr/bin/ruby /home/user/site/script/backgroundrb start" stop = "/usr/bin/ruby /home/user/site/script/backgroundrb stop" if cpu > 90% for 2 cycles then restart if totalmem > 512 MB for 2 cycles then restart if 4 restarts within 4 cycles then timeout group backgroundrb I have a non custom location for my some of my gems like packet (/home/ user/.gem/ruby/1.8/bin), and I suspect that part of the problem is that monit uses a spartan PATH (To quote http://mmonit.com/monit/documentation/monit.html - PATH=/bin:/usr/bin:/sbin:/usr/sbin), rendering these gems unreachable. Also, I get no errors in the backgroundrb debug log. I'd love to hear from anyone who has any idea how I might adjust backgroundrb (or my start/stop commands) in order to make this work. Cheers, Vaughan. From gethemant at gmail.com Mon Aug 10 22:16:59 2009 From: gethemant at gmail.com (hemant) Date: Tue, 11 Aug 2009 07:46:59 +0530 Subject: [Backgroundrb-devel] Question about backgroundrb and resource usage In-Reply-To: References: <80814BA8-6466-4A82-8E52-A3F6FCDCA239@thewilliams.ws> Message-ID: It totally depends on what you are doing. If your task is thread-safe you can even use thread_pool.defer() which will make sure that at least IO sensitive tasks are running concurrently. You don't need 10 workers there. One worker with some thread pool thrown around should suffice. On Tue, Aug 11, 2009 at 5:47 AM, Peter Gengler wrote: > You know that's an interesting point about the single worker; has anyone ever used Skynet? ?Their fortay is distributing map/reduce problems in the background, but they also mixed in a function to ActiveRecord called send_later which allowed you to send off any method of a model for later processing. ?I wrote my own mixin that does the same thing with Backgroundrb and a GenericWorker class (and I added send_now_or_later so I could have a fallback) because I liked keeping the functions on the model. ?That would lend itself to reducing the backgroundrb overhead as you'd have a single worker for any asynchronous AR processing, although it doesn't help with the cron-style jobs. > > \Peter > > -----Original Message----- > From: Adam Williams [mailto:adam at thewilliams.ws] > Sent: Monday, August 10, 2009 6:50 PM > To: Peter Gengler > Cc: backgroundrb-devel at rubyforge.org > Subject: Re: [Backgroundrb-devel] Question about backgroundrb and resource usage > > On Aug 10, 2009, at 3:44 PM, Peter Gengler wrote: > >> So I have a somewhat general question about backgroundrb and >> resource usage. ?I hadn't really considered the fact that it spools >> up a new rails process for each worker until I started making more >> workers; now that I'm have half a dozen it seems like the current >> design is really unscalable. ?I.e. if my rails stack takes 200MB of >> memory for example, that means 10 workers use 2GB of memory? ?That >> seems overly excessive to me, am I missing something that prevents >> that resource bloat? > > As far as I have seen, you're not missing anything. I had the same > problem and ended up having a single worker, with different methods > for the work. I'm considering getting rid of backgroundrb altogether, > favoring instead simply loading ActiveRecord and the model classes I > need, with the hope that a cron task can fire the thing up within a > few seconds. We need tasks to run every minute, so it must be fast > loading. Anyway, that would allow us to unload a couple hundred meg at > Engine Yard. > > ? adam > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org