Tutorial Robots.txt Means :

first fly

New Member
robots.txt

Robots.txt is one way of telling the Search Engine Bots about the web pages on your website which you do not want them to visit.

Robots.txt is useful for preventing the indexation of the parts of any online content that website owners do not want to display.

IF you want to block all search engine robots from crawling your website, just put the follow code:

User-agent: *
Disallow: /

IF you want to block Google from crawling your website, just put the follow code:

User-agent:
Disallow: /

It is important to decide the location of Robots.txt very carefully, or else errors might occur while displaying the website.
 

Saravanan28

Yellow Belt
Robots.txt file is very important. Using a robots.txt file and with a disallow direction, we can restrict bots or search engine crawling program from websites and or from certain folders and files.
 

rajdeep7830

Well-Known Member
Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

Allow indexing of everything

User-agent: *
Disallow:

Disallow indexing of everything

User-agent: *
Disallow: /
 
Top