MAXIMIZE

How to Exclude Pages from Getting Indexed

If there are pages or directories on your website that you do not want search engines to index you can exclude them using one of two different methods:

• Robots.txt
• META-robots code

Robots.txt

Robots.txt is a file that you place in the root of your web server. The file uses a simple syntax to exclude specific types of users - in this case search engine spiders - from parts of your website. You can either exclude specific search engine spiders or all spiders. To exclude all search engine spiders from all directories on your web server you write it like this:

User-agent: *
Disallow: /

Note: This would disallow everything including your home page!

Learn more about how to write robots.txt files at SearchTools.com [http://www.searchtools.com/robots/robots-txt.html].

TIP:
We recommend that you validate your robots.txt file before uploading it. There is no way to predict how a search engine will interpret a robots.txt file with errors. You can use the free validator at Search Engine World.

META-robots

META-robots are small pieces of code you can place in the header of your HTML-documents. You can use META-robots tags if you don't have access to your web server's root or if you want to exclude single pages on your website. You can read more about how to use the META-robots code at www.searchtools.com/robots/robots-meta.html

Back to Maximize