[Rake-devel] timestamp checking is slow?

Ittay Dror ittay.dror at gmail.com
Wed Jul 23 05:42:32 EDT 2008


More dicoveries:
It looks like 'File.exists' is slow. I did two changes:
- comment the first line in FileTask#needed?
- change FileTask#timestamp to:
        begin
                File.stat(name).mtime
        rescue
                Rake::EARLY
        end

this shaved 0.5 second (20% from the whole run time, or 30% from the 
time added after adding the header dependencies).

ittay

Ittay Dror wrote:
> I applied your change and it sped up the execution by 0.1s :-(.
>
> I then just added a 'return Rake::EARLY' at the start of #timestamp. 
> This also didn't improve time. So it must be something with the rake 
> code that calls the timestamp method.
>
> Ittay
>
> Mark Watson wrote:
>> I ran into a similar issue, the way i got around it was by caching the
>> timestamp the first time File.mtime is called.  This brings about its
>> own problems because you don't necessarily know that the file won't be
>> modified outside the file task.  If you have a task that generates
>> several files the cached timestamps won't be updated.
>>
>> I tried using Windows API directly instead of File.mtime but this
>> wasn't much faster.  I didn't get a chance to look into what make is
>> doing, but i suspect it is caching the timestamps too.  Make does
>> allow you to specify several files as the output of one task, possibly
>> as a way around the problems with caching the timestamp.
>>
>> # File task with timestamp speedup.  The regular file task will query
>> # the File.mtime for every task that depends on this file.  Typically
>> # when compiling c/c++ some header files are included very often, and
>> # if a a regular file task is used to represent header files then the
>> # File.mtime function will be called each time a source file depends
>> # on the header.  By caching the timestamp we insure that this
>> # operation occurs only once for each header file.  On typical c/c++
>> # projects where most files depend on a core set of headers, a ~3x
>> # speedup on rake startup time can be expected with this patch.
>> #
>> # WARNING: Caching the timestamp may cause problems if the file is
>> # generated/updated by another task as a side effect.
>> #
>> class Rake::FastFileTask < Rake::FileTask
>>   # Time stamp for file task.
>>   def timestamp
>>     # Cache the timestamp since it is accessed often
>>     if instance_variable_defined?(:@timestamp)
>>       @timestamp
>>     else
>>       # exist? and mtime can be done together with one call to File.stat, but
>>       # for some reason this takes longer on Windows.  Ruby is probably making
>>       # several Win32 calls.  Opimization would be to call GetFileAttributesEx()
>>       # directly.  Using Win32API module this is slightly faster, making the
>>       # call via a c extension would probably be faster again.
>>       if File.exist?(name)
>>         @timestamp = File.mtime(name.to_s)
>>         # Update the time stamp when the task is executed
>>         enhance do
>>           @timestamp = Time.now
>>         end
>>         @timestamp
>>       else
>>         # Build me!
>>         Rake::EARLY
>>       end
>>     end
>>   end
>> end
>>
>> 2008/7/22 Ittay Dror <ittay.dror at gmail.com>:
>>   
>>> Hi,
>>>
>>> I'm using Rake (actually Buildr) to compile C++ files. As part of that I'm
>>> loading dependencies on header files. The total amount of  dependencies is
>>> 869 on 142 obj files.
>>>
>>> The issue I'm seeing is that it takes a lot of time for Rake to timestamp
>>> these files to reach a conclusion that it has nothing to do. Compared to
>>> make that takes 0.7 seconds to run, Rake takes 3 seconds. If I remove the
>>> creation of the dependencies (but still leave the loading and parsing of the
>>> files) rake takes 1 second.
>>>
>>> I understand that 2 seconds seem like not a long time, but the project I'm
>>> working on has over 1000 of source files, so I expect the time to be much
>>> larger. Do you think anything can be done to make Rake closer to make? note
>>> that without the timestamp checking, the rest of the work doesn't amount to
>>> a lot of overhead, so I am hoping it is not the dynamic nature of Ruby that
>>> is the cause of the problem (but maybe poor implementation of
>>> File(file).mtime?)
>>>
>>> Thank you,
>>> Ittay
>>>
>>> --
>>> --
>>> Ittay Dror <ittay.dror at gmail.com>
>>>
>>>
>>> _______________________________________________
>>> Rake-devel mailing list
>>> Rake-devel at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/rake-devel
>>>
>>>     
>> _______________________________________________
>> Rake-devel mailing list
>> Rake-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/rake-devel
>>   
>
> -- 
> --
> Ittay Dror <ittay.dror at gmail.com>
>   

-- 
--
Ittay Dror <ittay.dror at gmail.com>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/rake-devel/attachments/20080723/091f9f85/attachment-0001.html>


More information about the Rake-devel mailing list