The Ruby Robots.txt Parser parses robots.txt files and allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Allow:, Sitemap: and Crawl-delay: rules.


This project has not yet categorized itself in the Trove Software Map.

Registered: 2008-07-28 19:34
Activity Percentile: 0%
View project activity statistics.

Developer Info
Project Admins:
Alex Dunae
Developers:
1 [View Members]
 

 

Latest File Releases
Package Version Date Notes / Monitor Download
This Project Has Not Released Any Files
 

Public Areas
Home Page Project Home Page
TrackerTracker

 - Bugs ( 0 open /0 total )
Bug Tracking System

 - Support Requests ( 0 open /0 total )
Tech Support Tracking System

 - Patches ( 0 open /0 total )
Patch Tracking System

 - Feature Requests ( 0 open /0 total )
Feature Request Tracking System

Forums Public Forums ( 2 messages in 2 forums )
Docs DocManager: Project Documentation
Lists Mailing Lists ( 0 public mailing lists)
Tasks Task Manager
  - To Do
  - Next Release
Surveys Surveys ( 0 surveys )
SCM SCM Repository (SVN: 0 updates, 0 adds)
 

  
Latest News
No News Items Found