fb16dfecae
Commit b7f05445c0 has added WWW entries to port Makefiles based on
WWW: lines in pkg-descr files.
This commit removes the WWW: lines of moved-over URLs from these
pkg-descr files.
Approved by: portmgr (tcberner)
2 lines
127 B
Text
2 lines
127 B
Text
WWW::RobotRules::Parser allows you to simply parse robots.txt files
|
|
as described in http://www.robotstxt.org/wc/norobots.html.
|