If you don’t know how or don’t want to create a robots.txt file an easier method for restricting access to your files is to use the robots meta tag. It prevents a page’s contents from being listed in the Google web index even if other sites link to it. Keep in mind this only applies to the Google robot (Googlebot) not other search engine robots.
How to create a robots meta tag
All meta tags are placed in the HEAD section of an HTML page as shown here:
<head>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>
The “noindex†value tells the search engines not to include this page in their listings. This doesn’t prevent the pages from being spidered because search engines have to keep revisiting the page in order to see if the tag is removed. The tag only keeps the page out.
Valid values that can be included
<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">
<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
Common uses of the Robots Meta Tag
1. “NOINDEX, FOLLOW” allows the links on that page to be followed, even though the page is not to be indexed.
<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">
2. “INDEX, NOFOLLOW” allows the page to be indexed but doesn’t allow the links to be followed.
<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">
3. “NOINDEX, NOFOLLOW” neither the page or the links on that page will be followed or indexed.
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
Important Note
Don’t confuse “NOFOLLOW” with the REL=”NOFOLLOW” link attribute e.g.
<a rel="nofollow" href="http://www.link.com">link text</a>
When a search engine bot comes across this link, it won’t crawl the site on the other end. Also, this link doesn’t count toward the PageRank of the site that is receiving it. Other search engines (besides Google) handle “NOFOLLOW” in slightly different ways.
Meta Tag Generator
http://www.submitcorner.com/cgi-bin/tools/metagen.cgi
If you’re worried about making a mistake when creating the robots meta tag use this meta tag generator tool.
Tip
The search engines will naturally index your pages and follow your links even if you don’t include the robots meta tag. That’s their main purpose.
If you don’t want a specific page indexed by a search engine then use the robots meta tag ie
Resources
Blocking Spiders, Cached Pages and More
Related Articles
What is a robots.txt file
How to Optimize Your Website for the Search Engines
301 vs 302 Redirect
Speak Your Mind