The Robots Exclusion Protocol (robots.txt) is a text file that tells certain search engine spiders not to index certain pages or sections of your website. For instance you may not want search engines to spider your site while it’s under construction or want to prevent a directory from being indexed.
What is a robots txt file used for?
