[Backgroundrb-devel] worker process hanging around, spiking memory

[hemant]

On Thu, Feb 28, 2008 at 5:54 AM, Clint Troxel <clint at ctro.net> wrote:
> Greg,
>
>  that is helpful.  thanks!
>
>  calling exit() from the workers does indeed kill the process.
>
>  Two more questions:
>
>  1) before the process is killed by exit() I repeatedly see the memory
>  of the bdrb process spike from 18 to 45 M -- is there an explanation
>  for this?  Also, the process seems to take a long time to finish --
>  minutes just to send 4 emails.  Could some sort of configuration be
>  causing behavior like this?

I do not think so. But in each process full rails environment is
loaded and hence its costly.
Thats why, you should probably just use thread_pools rather than
starting new processes.

>
>  2) More importantly, what is the recommended version to run?  And, how
>  do I check the version in my vendor dir? (I can't remember what
>  version you were at when I imported into my rails proj)
>

Recommended version to run is one from git mainline repo:

http://gitorious.org/projects/backgroundrb/repos/mainline

We are getting ready for release 1.0.3 and I have put together new
documentation at:

http://backgroundrb.gnufied.org

Please note that above location is just temporary and in a day or two
when 1.0.3 comes out, main documentation at rubyforge will reflect the
same doco.

>
>
>  On 2/27/08, Greg Campbell <gtcampbell at gmail.com> wrote:
>  > Hi Clint,
>  >
>  >
>  > On Wed, Feb 27, 2008 at 3:26 PM, Clint Troxel <clint at ctro.net> wrote:
>  > > Hi. Posted a general question earlier today.  I have a specific
>  > > question this time -- hoping someone can help out.  I think I'm
>  > > probably doing something incorrectly.  Here goes:
>  > >
>  > > I have a Worker method that sends a lot of emails.  The method looks like
>  > this:
>  > >
>  > >
>  > > def send_participant_reminders(args=nil)
>  > >    args[:emails].each do |email|
>  > >
>  > >      unless email.notified?
>  > >        Emailer.deliver_survey_reminder(email)
>  > >      end
>  > >
>  > >    end
>  > >  end
>  > >
>  > >
>  > > I call that worker from a controller like so:
>  > >
>  > >    MiddleMan.new_worker(:worker => :participant_worker)
>  > >    MiddleMan.ask_work(:worker => :participant_worker,
>  > >      :worker_method => :send_participant_reminders,
>  > >      :data => {:emails => @emails})
>  > >
>  > > ----------------------------
>  > >
>  > > My emails are being sent, but I'm seeing a new process created *each
>  > > time I invoke the code in the controller*.  That process starts out at
>  > > about 17m of memory then, once all work appears to have been
>  > > completed, jumps to about 50m.  The process stays around until
>  > > "script/backgroundrb stop".
>  > >
>  > > Maybe I've been reading the examples incorrectly?
>  > >
>  > > Any help would be appreciated.  Seeing this same problem (I think)
>  > > suck up 4G of ram on a production server.  yuck.
>  > >
>  > > Thanks again,
>  > > clint
>  >
>  > Calling new_worker does indeed create a new backgroundrb process (which may
>  > or may not be necessary in your case, see below).  If you do in fact need to
>  > create a new process for each task, you'll need to call either exit (from
>  > within the worker) or MiddleMan.delete_worker (from the controller) once the
>  > task is done. If you need to keep track of these worker tasks while they're
>  > running (to call delete_worker from the controller, or to use ask_status),
>  > you should also use a job_key when you create it.
>  >
>  > You may not actually need the new process, however.  If the likely use case
>  > is that there will only be one of these tasks running at a time, then you
>  > can just omit the new_worker call entirely and direct all calls to a single
>  > running instance of the worker process.  You may also want to investigate
>  > using thread_pool.defer inside the worker to run the individual tasks in
>  > threads (rather than spawning new worker processes for each one) if it seems
>  > likely that there will be multiple requests running simultaneously.
>  >
>  > Thanks,
>  > Greg
>  > p.s. If I've misrepresented anything, I'd like to invite Hemant or anyone
>  > else more knowledgeable than I am to correct me, as I've only started
>  > working on backgroundrb in the last few weeks.
>  >

You have been absolutely correct!