The meta robots tag is not one of the common tags to include in your web pages. I decided to mention it here because you see it used on some sites when it has a clear purpose.
All web sites will naturally get indexed by the search engines without using this tag.
The code for the robots meta tag must be included within the header tags of your web pages.
Top 7 Benefits and Uses of the Robots Meta Tag
1. Tell search engines not to index your pages (noindex)
<meta name="robots" content="noindex">
2. Tell search engines to index your site but not to follow links (nofollow)
Use this code for pages which contain useful content but have links which are irrelevant or obsolete.
<meta name="robots" content="nofollow">
3. Tell search engines to not index or follow links on your web page (noindex,nofollow)
If you include this code in the header of every page of your web site, it will not be indexed.
<meta name="robots" content="noindex,nofollow">
4. Tell search engines to index your site and follow links (index,follow)
<meta name="robots" content="index,follow">
Because search engines index and follow links naturally, there is no need to add this tag.
5. Tell search engines not to cache your pages (noarchive)
<meta name="robots" content="noarchive">
6. Prevents a description appearing below the page in the search results and prevents page caching (nosnippet)
<meta name="robots" content="nosnippet">
7. Prevent search engines from displaying DMOZ data in the search results of your site (noodp)
<meta name="robots" content="noodp">
The above code prevents all search engines from using this information for the page’s description.
To specifically prevent Google from using this information for a page’s description, use:
<meta name="googlebot" content="noodp">
TIPS
A. A word of warning about using content value “NONE”
<meta name="robots" content="none">
Some webmasters use content value “NONE” to indicate no robots restrictions, however they unknowingly block all search engines from indexing the content on their web pages.
B. If you add the robots meta tags to a framed site, make sure you include them on both the FRAMESET and the FRAME pages.
C. Rather than use the meta robots tag to not index a web page use the robots.txt file to block crawler access to pages.
What is robots.txt file?
Conclusion
If you don’t specific the robots meta Tag on a web page, the spiders will index that page and all its links by default. Only use this tag when you don’t want certain parts of your web page indexed.
Resources
Yahoo Provides NOYDIR Opt-Out Of Yahoo Directory Titles & Descriptions
http://searchengineland.com/070228-140603.php
thats perfect info!
Hi
How do i implement the
on a drupal cms website
Moh..Sorry I’ve never used Druplal. Ask on their forums . Post your question here with more details if you don’t get a clear answer.
Thanks! Great explaination about robot meta tag.
Hey, very informative and interesting. Looks like you put a lot of effort and research into it. Keep up the good work!!
Nice Post. Really an informative one for beginners and pro.
Nice sharing.. Eventually the major search engines began discontinuing the use of Meta tags as major criteria for listing sites. Google had always ignored the use of Meta tags, and currently will only index Google Meta Tags.