[Backgroundrb-devel] Hello to the group

Chad Thatcher chad.thatcher at gmail.com
Sat Jan 24 07:35:05 EST 2009


Thanks Hemant, you are absolutely correct about threads being too much  
overhead for what I need so this looks like a perfect solution!

Cheers,

Chad.


On 24 Jan 2009, at 01:45, hemant wrote:

> On Sat, Jan 24, 2009 at 1:46 AM, Chad Thatcher <chad.thatcher at gmail.com 
> > wrote:
>> Thanks Hemant,
>>
>> Yes I was specifically trying to avoid threads.  However, I am  
>> guessing I
>> could configure it so that only one thread ran at one time right?   
>> That
>> would cut down on the complexity.
>
> Yes, changing thread pool size to 1 usually helps. Libraries like
> mechanize (even Net/HTTP) crap out under concurrent threads but if
> your pool size is 1, you won't have these problems, thread pool will
> be just a queue. But if you don't want any concurrency, why not just
> do something like this:
>
> class FooWorker < BackgrounDRb::MetaWorker
>  def create(args = nil)
>    @task_array = []
>     # check task array for pending tasks and run them
>     add_periodic_timer(2) { run_from_task_array() }
>  end
>
>  def accept_indexing(args = nil)
>     klass = Struct.new(:cmd_args)
>     # add task to the task array
>     @task_array << klass.new(args)
>  end
>
> def run_from_task_array
>     task = @task_array.shift
>     # run the task
>  end
>
>   def check_count
>      return @task_array.count
>   end
> end
>
> if you don't want threads and above code will work too. Specially
> since thread pool uses Queue data structure from ruby stdlib which is
> quite heavyweight and thread safe, but you don't want any thread
> safety here, so why pay additional penalty for that? All your tasks
> run in same main thread.
>
>
>
>>
>> Is the enq_xxx stuff only time based?  If so I wonder if it would be
>> difficult to add a feature that allows you to add a whole bunch of  
>> tasks to
>> the queue and then they just run one after the other in the order  
>> in which
>> they were added.  Maybe I could intercept persistent_job#finished!  
>> to find
>> the next (grouped) task in the queue and update it to start a few  
>> seconds in
>> the future so the bgrb picks it up and runs it etc.  But that seems  
>> evil
>> somehow :).
>
> You can totally do that! Check add_timer or add_periodic_timer
>
>>
>> I realise this might not be the target usage for backgroundrb but I  
>> look
>> forward to hearing your thoughts because I'm hoping it won't be too
>> difficult for me to do.  The ultimate fall-back, of course, is to  
>> roll my
>> own task queue that uses a table similar table to bdrb_job_queues  
>> but that
>> seems wasteful when there is so much there for it already.
>>
>
> Right.
>
>>
>> On 23 Jan 2009, at 18:57, hemant wrote:
>>
>>> That data is in write Queue of master worker. It can't be  
>>> meaningfully
>>> extracted to provide how many tasks are in the queue. Put it other
>>> way, its not even in a Queue in traditional sense. Data is just  
>>> there
>>> in a buffer so that master will write it to the socket when socket  
>>> is
>>> ready for write.
>>>
>>> If you want Queue. You will have either use enq_xxx stuff or  
>>> provide a
>>> sugar around asyc_index() method so as actual method is not directly
>>> called but is added to the thread pool (thread_pool.defer), this way
>>> you can query how many tasks are there in thread pool queue.
>>>
>>> However word "thread" comes with its own meaning and associated
>>> crapiness in ruby, so beware.
>>>
>>>
>>> On Fri, Jan 23, 2009 at 6:35 PM, Chad Thatcher <chad.thatcher at gmail.com 
>>> >
>>> wrote:
>>>>
>>>> Hello everyone.
>>>>
>>>> I am making several asynchronous calls to one particular method  
>>>> in my
>>>> worker
>>>> like this:
>>>>
>>>> MiddleMan.worker(:reindex_worker).async_reindex(:arg => options)
>>>>
>>>> If I call this several times in a row, all subsequent calls are  
>>>> put in a
>>>> queue and I would like to be able to keep track of this.  The  
>>>> behavior is
>>>> perfect because I only ever want reindex() running once at any  
>>>> one time
>>>> but
>>>> I would like to be able to show my users where in the queue their  
>>>> request
>>>> is.  Is there any way to view what async calls are lined up?  The
>>>> worker_info stuff doesn't seem to give away any details like this.
>>>>
>>>> I also thought about using "enq_" to schedule but is there a way to
>>>> specify
>>>> that the trigger is the completion of the previous scheduled  
>>>> instance and
>>>> so
>>>> on?  Or is the scheduling purely time based?
>>>>
>>>> Kind regards,
>>>>
>>>> Chad.
>>>>
>>>> _______________________________________________
>>>> Backgroundrb-devel mailing list
>>>> Backgroundrb-devel at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>>>
>>>
>>>
>>>
>>> --
>>> Let them talk of their oriental summer climes of everlasting
>>> conservatories; give me the privilege of making my own summer with  
>>> my
>>> own coals.
>>>
>>> http://gnufied.org
>>
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>
>
>
>
> -- 
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org



More information about the Backgroundrb-devel mailing list