Help Me/Question What is robots.txt?


White Belt
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
Robots.txt is a text file generated for search engine crawlers. When search engine crawler will came to your site then they will first access your Robots.txt while and then will see that which part of your site has to crawl. In simple words, robots.txt file tells search cralwers which part of your site, you have to crawl.


Yellow Belt
The robots exclusion standard, also known as the robots exclusion protocol or robots.txt protocol, is a standard used by websites to communicate with web crawlers and other web robots.


Yellow Belt
Robots.txt file allows spiders or crawlers to allow or disallow to crawl all pages of a website or a particular webpage. Robots.txt file is a simple text file that must be placed in root directory of a website.

Zirkon Kalti

Content Writer
Robots.txt, also known as Robots Exclusion Protocol (REP), is a text file that contains codes that tells the search engine how to crawl your site. You have to upload the robots.txt to the top level directory for example in order for the search engine to take into consideration.

Manish Mishra

Content Writer
In an easy understanding, robots.txt gives access to Web Crawler how it should be used in World Wide Web. This gives command to crawler how an end user would see our website.


Well-Known Member
I was reading about your comments and your definitions about robot.txt. all of your definitions about robot.txt is correct. Because of reading about how you define it, I also get the idea on how to use it properly.

steve taylor

White Belt
A robots.txt file is a text file at the root of your site that indicates those parts of your site you do not want accessed by search engine web crawlers. The file uses the robots which is a set of rules with a small set of commands that can be used to point access to your site by section and by specific kinds of web crawlers.

Swati Mishra

Content Writer
It is a small notepad file with a few line of code that strictly allows or disallows search engines to notify which pages need to be crawled and which to be complete ignored.


Well-Known Member
robots.txt is used to instruct the search engine spiders to crawl which pages and also about the webpages that should not be crawled.


Money Making Ideas Online UAE, UK, USA
Search engines send out tiny programs called spiders or robots to search your site and bring information back so that your pages can be indexed in the search results and found by web users. If there are files and directories you do not want indexed by search engines, you can use the "robots.txt" file to define where the robots should not go.
Robot.txt is a text file that is used for instruct to crawler the web page is crawling or not. If you want to not crawl any specific webpage or folder in your directory than you instruct to crawler through out the file.