be1be4cd15
Protego is a pure-Python robots.txt parser with support for modern conventions. It is mainly used by Scrapy application when crawling websites. WWW: https://pypi.org/project/Protego/
4 lines
184 B
Text
4 lines
184 B
Text
Protego is a pure-Python robots.txt parser with support for modern conventions.
|
|
It is mainly used by Scrapy application when crawling websites.
|
|
|
|
WWW: https://pypi.org/project/Protego/
|