ad4f267b7a
search.cpan.org is shutting down. It will redirect to metacpan.org after June 25, 2018. With hat: perl
6 lines
303 B
Text
6 lines
303 B
Text
This module parses /robots.txt files which are used to forbid conforming
|
|
robots from accessing parts of a web site. The parsed files are kept in
|
|
a WWW::RobotRules object, and this object provides methods to check if
|
|
access to a given URL is prohibited.
|
|
|
|
WWW: https://metacpan.org/release/WWW-RobotRules
|