Release Name: 0.2.0
Notes:
A Ruby web-spider framework for easily developing programs to crawl a domain and gather useful statistics.
Changes:
* Option to obey robots.txt (:obey_robots_txt => true)
* Tracks page depth and referer during crawl
* Send referer header with HTTP requests
* Option to limit crawl depth (:depth_limit => n)
|