Skip Navigation

Scout Archives

Home Projects Publications Archives About Sign Up or Log In

The Web Robots Pages

Martin Koster, a software engineer working in the United Kingdom, maintains this website about Web robots. Web robots, also known as Web Wanderers, Crawlers, or Spiders, are "programs that traverse the Web automatically." The website provides information on how Web robots operate, how they feed search engines, and ways to steer them away from (or to) your pages. Also available from this website is a database of currently known robots, with descriptions and contact details. For more information on technical aspects of designing, building, and operating Web robots, visitors can read discussions from an archived mailing list, browse some older articles and papers posted, or follow links to related websites.
Archived Scout Publication URL
Scout Publication
Creator
Publisher
Language
Date of Scout Publication
April 22nd, 2005
Date Of Record Creation
April 21st, 2005 at 11:34am
Date Of Record Release
June 21st, 2005 at 3:14pm
Resource URL Clicks
3

Internal

Cumulative Rating
0
Add Comment

Comments

(no comments available yet)