Help Me/Question Where I Can Submit Robot.txt File?

niranjan kumar

White Belt
As you want to know where you can put, first you need to know about Robots.txt. It is used for crawler and spiders, Because search engine understand better you website. Robots.txt not easy to make it's need some experiences for making robots.txt.
You put robots.txt on your server home directory.


White Belt
Robots.txt file is placed in your www or public_html directory. And you can Download your edited robots.txt code from the robots.txt Tester page by clicking Download in the Submit dialog. Upload your new robots.txt file to the root of your domain as a text file named robots.txt (the URL for your robots.txt file should be /robots.txt).


Well-Known Member
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.


Red Belt
Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.


The robots.txt file should be located in the root directory of the site, but its presence on the site is not mandatory. It is designed to determine which pages of the site should exclude their processing and indexing by search engines. Or specify which search engine robots should or should not index the site. A standard robots.txt file might look like this:
User-agent: *
Disallow: / pages /
Sitemap: http: //yoursite/sitemap.xml
The directive User-agent: * means that all robots are allowed to visit the site.
The Disallow directive allows you to specify which pages to exclude from indexing. In the directive Host you specify the main address of your site. In addition, you can specify a sitemap file in this file.


Well-Known Member
The robots.txt file must be located at the root of the website host that it applies to. And If you have a WordPress Site then you need to install Yoast plugin and connected the Google search console.