Files | Admin

Notes:

Release Name: 0.2.0

Notes:
A Ruby web-spider framework for easily developing programs to crawl a domain and gather useful statistics. 



Changes: * Option to obey robots.txt (:obey_robots_txt => true) * Tracks page depth and referer during crawl * Send referer header with HTTP requests * Option to limit crawl depth (:depth_limit => n)