PyMOTW: robotparser

robotparser implements a parser for the robots.txt file format,
including a simple function for checking if a given user agent can
access a resource. It is intended for use in well-behaved spiders or
other crawler applications that need to either be throttled or otherwise
restricted.

Read more at pymotw.com: robotparser