[Backgroundrb-devel] Hello to the group

hemant gethemant at gmail.com
Fri Jan 23 20:45:07 EST 2009


On Sat, Jan 24, 2009 at 1:46 AM, Chad Thatcher <chad.thatcher at gmail.com> wrote:
> Thanks Hemant,
>
> Yes I was specifically trying to avoid threads.  However, I am guessing I
> could configure it so that only one thread ran at one time right?  That
> would cut down on the complexity.

Yes, changing thread pool size to 1 usually helps. Libraries like
mechanize (even Net/HTTP) crap out under concurrent threads but if
your pool size is 1, you won't have these problems, thread pool will
be just a queue. But if you don't want any concurrency, why not just
do something like this:

class FooWorker < BackgrounDRb::MetaWorker
  def create(args = nil)
    @task_array = []
     # check task array for pending tasks and run them
     add_periodic_timer(2) { run_from_task_array() }
  end

  def accept_indexing(args = nil)
     klass = Struct.new(:cmd_args)
     # add task to the task array
     @task_array << klass.new(args)
  end

 def run_from_task_array
     task = @task_array.shift
     # run the task
  end

   def check_count
      return @task_array.count
   end
end

if you don't want threads and above code will work too. Specially
since thread pool uses Queue data structure from ruby stdlib which is
quite heavyweight and thread safe, but you don't want any thread
safety here, so why pay additional penalty for that? All your tasks
run in same main thread.



>
> Is the enq_xxx stuff only time based?  If so I wonder if it would be
> difficult to add a feature that allows you to add a whole bunch of tasks to
> the queue and then they just run one after the other in the order in which
> they were added.  Maybe I could intercept persistent_job#finished! to find
> the next (grouped) task in the queue and update it to start a few seconds in
> the future so the bgrb picks it up and runs it etc.  But that seems evil
> somehow :).

You can totally do that! Check add_timer or add_periodic_timer

>
> I realise this might not be the target usage for backgroundrb but I look
> forward to hearing your thoughts because I'm hoping it won't be too
> difficult for me to do.  The ultimate fall-back, of course, is to roll my
> own task queue that uses a table similar table to bdrb_job_queues but that
> seems wasteful when there is so much there for it already.
>

Right.

>
> On 23 Jan 2009, at 18:57, hemant wrote:
>
>> That data is in write Queue of master worker. It can't be meaningfully
>> extracted to provide how many tasks are in the queue. Put it other
>> way, its not even in a Queue in traditional sense. Data is just there
>> in a buffer so that master will write it to the socket when socket is
>> ready for write.
>>
>> If you want Queue. You will have either use enq_xxx stuff or provide a
>> sugar around asyc_index() method so as actual method is not directly
>> called but is added to the thread pool (thread_pool.defer), this way
>> you can query how many tasks are there in thread pool queue.
>>
>> However word "thread" comes with its own meaning and associated
>> crapiness in ruby, so beware.
>>
>>
>> On Fri, Jan 23, 2009 at 6:35 PM, Chad Thatcher <chad.thatcher at gmail.com>
>> wrote:
>>>
>>> Hello everyone.
>>>
>>> I am making several asynchronous calls to one particular method in my
>>> worker
>>> like this:
>>>
>>> MiddleMan.worker(:reindex_worker).async_reindex(:arg => options)
>>>
>>> If I call this several times in a row, all subsequent calls are put in a
>>> queue and I would like to be able to keep track of this.  The behavior is
>>> perfect because I only ever want reindex() running once at any one time
>>> but
>>> I would like to be able to show my users where in the queue their request
>>> is.  Is there any way to view what async calls are lined up?  The
>>> worker_info stuff doesn't seem to give away any details like this.
>>>
>>> I also thought about using "enq_" to schedule but is there a way to
>>> specify
>>> that the trigger is the completion of the previous scheduled instance and
>>> so
>>> on?  Or is the scheduling purely time based?
>>>
>>> Kind regards,
>>>
>>> Chad.
>>>
>>> _______________________________________________
>>> Backgroundrb-devel mailing list
>>> Backgroundrb-devel at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>>
>>
>>
>>
>> --
>> Let them talk of their oriental summer climes of everlasting
>> conservatories; give me the privilege of making my own summer with my
>> own coals.
>>
>> http://gnufied.org
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org


More information about the Backgroundrb-devel mailing list