PyMOTW: robotparser

robotparser implements a parser for the robots.txt file format, including a simple function for checking if a given user agent can access a resource. It is intended for use in well-behaved spiders or other crawler applications that need to either be throttled or otherwise restricted.

Read more at pymotw.com: robotparser