[Rubygems-developers] Detecting server changes

Jim Weirich jim at weirichhouse.org
Fri Nov 26 23:51:34 EST 2004

There was some discussion a while back on the efficiency of detecting changes 
on the server.  I was doing a little experimentation and got the following 
results ...

                 user     system      total        real
HEAD         0.020000   0.000000   0.020000 (  0.115111)
IMS(no)      0.000000   0.000000   0.000000 (  0.109897)
IMS(yes)     0.180000   0.020000   0.200000 (  1.952789)
NMod         0.160000   0.000000   0.160000 (  2.156582)
open-uri     0.290000   0.030000   0.320000 (  2.112810)
open-uri/sz  0.000000   0.000000   0.000000 (  0.146552)

HEAD        -- Just request the headers.
IMS(no)     -- Use If-Modified-Since (where the answer is no).
IMS(yes)    -- Use If-Modified-Since (where the answer is yes).
NMod        -- No IMS header (normal HTTP get)
open-uri    -- Using open-uri straight up
open-uri/sz -- Using open-uri will the break on content length.

All of the shortcuts (HEAD, IMS(no) and open-uri/sz) all take approximately 
the same amount of time, and all are much faster than downloading the entire 
compressed source.

Based on that, I don't think there is a strong compeling reason based on these 
benchmarks to switch from the open-uri/sz technique that is in use currently.  
(although the size technique can fail with non-monotonically increasing file 
sizes, in which case the IMS technique looks attractive).

But, there is a problem with the current code.  On many systems, the cache is 
in an area that can only be written while running under sudo.  That means the 
cache is updated on installs, but not on general queries.  This forces the 
gem command to download the yaml source on every command unless a recent 
install has occurred.  

I'm suggesting that we store the cached results in the users's home directory 
(or appropriate directory on windows).  This allows a gem list command to 
update the cache without running under sudo.  We could act intelligently and 
only store in the home directory if the normal cache directory is write 
protected.  We could also check both the system-wide cache and the 
user-specific cache when comparing against the server.

Any thoughts before I dive into this?

-- Jim Weirich    jim at weirichhouse.org     http://onestepback.org
"Beware of bugs in the above code; I have only proved it correct, 
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)

More information about the Rubygems-developers mailing list