People always ask what is robots.txt? It is simply a text file. It provides directives, or instructions, to search engine crawlers about which parts of your site may be crawled and which parts may not be crawled. By allowing search engines to crawl the content on your site, you are also allowing them to index the content from your site. Indexing allows your content to be visible in search results. (more…)