[skip to content](#content)
Advertisement
[](/)
Navigation
* [The /robots.txt](/robotstxt.html)
* [<META> tags](/meta.html)
* [Frequently Asked Questions](/faq.html)
* [Mailing list](/mailinglist.html)
* [Other Sites](/other.html)
* [About robotstxt.org](/about.html)
Tools
* [/robots.txt checker](/checker.html)
* [Robots Database](/db.html)
* [IP lookup](/iplookup.html)
Advertisement
The Web Robots Pages
--------------------
Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as [Google](http://www.google.com/) use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.
On this site you can learn more about web robots.
* [About /robots.txt](robotstxt.html) explains what /robots.txt is, and how to use it.
* The [FAQ](faq.html) answers many frequently asked questions, such as [How do I stop robots visiting my site?](/faq/prevent.html) and [How can I get the best listing in search engines?"](/faq/bestlisting.html)
* The [Other Sites](other.html) page links to external resources for robot writers and webmasters.
* The [Robots Database](db.html) has a list of robots.
* The [/robots.txt checker](checker.html) can check your site's /robots.txt file and meta tags.
* The [IP Lookup](iplookup.html) can help find out more about what robots are visiting you.
Advertisement
[About this site](/about.html) | [Privacy and cookies policy](/privacy.html) | [Contact us](/contact.html) | [© 2007. All rights reserved.](/copyrights.html) | Hosted by Mythic Beasts