can anyone help?

[mileycullens]

Please can anyone explain me what is robots.txt file and its correct format?

[DavidAmerland]

Yes, it's basically a .txt file created in Notepad, saved as robots.txt and uploaded to the root directory of your site. It is the official way to let the search engine bots (all major search engine bots respect it) know what to crawl and index and what not to and it was the primary means of protecting material.

These days it is necessary in order to help search engine crawlers save time. If you do a Google search for 'robots.txt' you will find the format you need to apply - it's pretty simple actually, about 2 - 3 lines unless you have some specific requirements on your site.

[julietegecy]

Web site owners use the /robots.txt file to give instructions about their site to web robots.
There are two important considerations when using /robots.txt:
1. robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
2. the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.
check on redalkemi site

[zbastian_19]

Robots.txt will allow you to control the crawlers by allowing them or not. I would say that it is wise to have this file inside the root of a website since SE looks for this file. This file will also allow you to see the and filter the spiders visit comes to your website. And there lots of reason why this file is needed.

[regmac]

It's basically intended for giving directive to the bots on how to handle a site when they visit them.