Martin Wilke
|
201a939790
|
WWW::RobotRules::Parser allows you to simply parse robots.txt files
as described in http://www.robotstxt.org/wc/norobots.html.
WWW: http://http://search.cpan.org/dist/WWW-RobotRules-Parser/
PR: ports/120160
Submitted by: Masahiro Teramoto <markun at onohara.to>
|
2008-02-16 13:52:51 +00:00 |
|