PyMOTW: robotparser

robotparser implements a parser for the robots.txt file format,
including a simple function for checking if a given user agent can
access a resource. It is intended for use in well-behaved spiders or
other crawler applications that need to either be throttled or otherwise

Read more at robotparser