9381cc3bb0
robot_detection is a python module to detect if a given HTTP User Agent is a web crawler. It uses the list of registered robots from http://www.robotstxt.org: (Robots Database)
3 lines
177 B
Text
3 lines
177 B
Text
robot_detection is a python module to detect if a given HTTP User Agent
|
|
is a web crawler. It uses the list of registered robots from
|
|
http://www.robotstxt.org: (Robots Database)
|