Martin Koster, a software engineer working in the United Kingdom, maintains this website about Web robots. Web robots, also known as Web Wanderers, Crawlers, or Spiders, are "programs that traverse the Web automatically." The website provides information on how Web robots operate, how they feed search engines, and ways to steer them away from (or to) your pages. Also available from this website is a database of currently known robots, with descriptions and contact details. For more information on technical aspects of designing, building, and operating Web robots, visitors can read discussions from an archived mailing list, browse some older articles and papers posted, or follow links to related websites.
Comments