A robots.txt, as the extension suggests, is a text file. It is used to inform the search engine bots or spiders about the pages in your website that they are not allowed to visit. In short it acts like a security guard which prevents the search engines from visiting the webpages you do not want them to visit.
Robots.txt and SEO
SEO or search engine optimization is performed on a website in order to ensure that the search engine bots can crawl all over the website and index the pages for the search engines. If the bots do not visit all the webpages then it will be difficult for the website to get a good rank in search engines through SEO. The primary function of a robots.txt file therefore goes against the benefits of SEO.
However, the reason why websites require a robots.txt file is because of privacy and security issues.
How Does Robots.txt Help In Website Privacy Issues?
An e-commerce website might have a database which contains the private information about the customers of the site including crucial bits of data like credit card information etc. This information is, of course, accessible to the website. A robots.txt file would have to be used to prevent search engines bots from visiting the webpages which contain such sensitive information. Since the bots are unable to access them, the pages are not indexed by the search engine.
If there was no robots.txt file, then the pages would have got indexed. Any person could have accessed those sensitive and personal bits of information through the search engines. This would have been a nightmare to say the least. Robots.txt files are used to prevent such scenarios.
Other Ways to Use Robots.txt
Apart from preventing the exposure of credit card information, the robots.txt file can prevent the search engines from accessing other private information. Some companies might allow their employees to conduct business conversations through a section on the company’s website. Important and sensitive information like ideas and business decisions might be passed between the employees through those pages. Some websites even have a wiki section for the employees to use. Such information might cause a lot of problems when viewed by outsiders. A robots.txt file will prevent the information from becoming public through the search engines.
Canonicalisation
Robots.txt can help prevent another common problem called canonicalisation. This problem is also known as a duplicate content problem but that nomenclature is incorrect. The problem of canonicalisation can occur when a single website contains multiple webpages that have the same data or information.
A particular page might have a separate version which is meant for printing purposes. The information on both the pages will be similar in all respects and this can become a problem for the bots of search engines. The bots will have the added task of deterring which of those pages is actually the canonical or the original one. A robots.txt file can be used so that the secondary versions of the pages are not indexed by the bots.