Tag Archives | robots.txt

The robots.txt file

Tweet The robots.txt is a file that can be used to restrict access to your site or parts of your site by preventing access from the search engine robots as they crawl the web. Most of the time this is used to prevent duplicate content (eg for a forum you do not want your printthread.php […]

Continue Reading Comments { 0 }