Directory of Robots Resources

Home > Computers > Internet > Searching > Search Engines > Robots

Web robots (also known as crawlers or spiders) are programs that traverse the Web automatically, and which are used by search engines to index the Web, or part of it.

Resources in This Category

• About Search Indexing Robots and Spiders

  http://www.searchtools.com/robots/
  Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.

• ACAP - Automated Content Access Protocol

  http://www.the-acap.org/
  Standard being developed on behalf of content publishers to communicate permissions information more extensively than is the case with robots.txt. Project documents, implementation and background information.

• Bots vs Browsers

  http://www.botsvsbrowsers.com/
  This large database lists user agents in categories and distinguishes between robots and browsers.

• HTTP User Agent Index

  http://www.siteware.ch/webresources/useragents/db.html
  An alphabetical list of user agents and the deployer behind them, compiled by Christoph Rüegg.

• List of User-Agents

  http://www.user-agents.org/
  A searchable database of user-agents with information about their type, purpose and origin.

• Search Engine IP Addresses

  http://www.iplists.com/
  Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.

• Search Engine Robots and Other User Agents

  http://www.jafsoft.com/searchengines/webbots.html
  John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.

• The Web Robots Pages

  http://www.robotstxt.org/
  Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.

• User Agent String

  http://user-agent-string.info/
  Tool from ASAP Consulting s.r.o. for detailed user agent string analysis using an online form. Includes databases of browsers and robots.

• User-Agents.My-Addr.com

  http://user-agents.my-addr.com/
  Contains a database of user-agents for crawlers, spiders, browsers; tools for user-agent lookup and tools for user-agent string search.

Related Categories

• Home > Computers > Internet > Searching > Search Engines > Sitemaps
• Home > Computers > Internet > Web Design and Development > Authoring > Online Tools > robots.txt
• Home > Computers > Software > Internet > Site Management > Log Analysis

 

Home > Computers > Internet > Searching > Search Engines > Robots

 

---

 

Thanks to DMOZ, which built a great web directory for nearly two decades and freely shared it with the web. About us