[Rake-devel] timestamp checking is slow?

Mark Watson watsonmw at gmail.com
Wed Jul 23 14:30:01 EDT 2008


I should say that I'm running on WindowsXP.  On WindowsXP I found
pretty much the opposite, minimizings calls to File.exist? provided a
tiny improvement, but the change to File.mtime provided a large
speedup :p

What OS are you running on?

Also it may depend on the project, i.e. the total number of headers
and how the dependencies are distributed among the source files.

2008/7/23 Ittay Dror <ittay.dror at gmail.com>:
> More dicoveries:
> It looks like 'File.exists' is slow. I did two changes:
> - comment the first line in FileTask#needed?
> - change FileTask#timestamp to:
>         begin
>                 File.stat(name).mtime
>         rescue
>                 Rake::EARLY
>         end
>
> this shaved 0.5 second (20% from the whole run time, or 30% from the time
> added after adding the header dependencies).
>
> ittay
>
> Ittay Dror wrote:
>
> I applied your change and it sped up the execution by 0.1s :-(.
>
> I then just added a 'return Rake::EARLY' at the start of #timestamp. This
> also didn't improve time. So it must be something with the rake code that
> calls the timestamp method.
>
> Ittay
>
> Mark Watson wrote:
>
> I ran into a similar issue, the way i got around it was by caching the
> timestamp the first time File.mtime is called.  This brings about its
> own problems because you don't necessarily know that the file won't be
> modified outside the file task.  If you have a task that generates
> several files the cached timestamps won't be updated.
>
> I tried using Windows API directly instead of File.mtime but this
> wasn't much faster.  I didn't get a chance to look into what make is
> doing, but i suspect it is caching the timestamps too.  Make does
> allow you to specify several files as the output of one task, possibly
> as a way around the problems with caching the timestamp.
>
> # File task with timestamp speedup.  The regular file task will query
> # the File.mtime for every task that depends on this file.  Typically
> # when compiling c/c++ some header files are included very often, and
> # if a a regular file task is used to represent header files then the
> # File.mtime function will be called each time a source file depends
> # on the header.  By caching the timestamp we insure that this
> # operation occurs only once for each header file.  On typical c/c++
> # projects where most files depend on a core set of headers, a ~3x
> # speedup on rake startup time can be expected with this patch.
> #
> # WARNING: Caching the timestamp may cause problems if the file is
> # generated/updated by another task as a side effect.
> #
> class Rake::FastFileTask < Rake::FileTask
>   # Time stamp for file task.
>   def timestamp
>     # Cache the timestamp since it is accessed often
>     if instance_variable_defined?(:@timestamp)
>       @timestamp
>     else
>       # exist? and mtime can be done together with one call to File.stat,
> but
>       # for some reason this takes longer on Windows.  Ruby is probably
> making
>       # several Win32 calls.  Opimization would be to call
> GetFileAttributesEx()
>       # directly.  Using Win32API module this is slightly faster, making the
>       # call via a c extension would probably be faster again.
>       if File.exist?(name)
>         @timestamp = File.mtime(name.to_s)
>         # Update the time stamp when the task is executed
>         enhance do
>           @timestamp = Time.now
>         end
>         @timestamp
>       else
>         # Build me!
>         Rake::EARLY
>       end
>     end
>   end
> end
>
> 2008/7/22 Ittay Dror <ittay.dror at gmail.com>:
>
>
> Hi,
>
> I'm using Rake (actually Buildr) to compile C++ files. As part of that I'm
> loading dependencies on header files. The total amount of  dependencies is
> 869 on 142 obj files.
>
> The issue I'm seeing is that it takes a lot of time for Rake to timestamp
> these files to reach a conclusion that it has nothing to do. Compared to
> make that takes 0.7 seconds to run, Rake takes 3 seconds. If I remove the
> creation of the dependencies (but still leave the loading and parsing of the
> files) rake takes 1 second.
>
> I understand that 2 seconds seem like not a long time, but the project I'm
> working on has over 1000 of source files, so I expect the time to be much
> larger. Do you think anything can be done to make Rake closer to make? note
> that without the timestamp checking, the rest of the work doesn't amount to
> a lot of overhead, so I am hoping it is not the dynamic nature of Ruby that
> is the cause of the problem (but maybe poor implementation of
> File(file).mtime?)
>
> Thank you,
> Ittay
>
> --
> --
> Ittay Dror <ittay.dror at gmail.com>
>
>
> _______________________________________________
> Rake-devel mailing list
> Rake-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rake-devel
>
>
>
> _______________________________________________
> Rake-devel mailing list
> Rake-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rake-devel
>
>
> --
> --
> Ittay Dror <ittay.dror at gmail.com>
>
>
> --
> --
> Ittay Dror <ittay.dror at gmail.com>
>
> _______________________________________________
> Rake-devel mailing list
> Rake-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rake-devel
>


More information about the Rake-devel mailing list