Welcome to WebmasterServe!

FREE TO JOIN! Join us now to engage in informative and friendly discussions about Webmastering, SEO, SEM, Internet Marketing, Programming, Graphic Design, Online Jobs and more. What are you waiting for? Ready to join our friendly community? It takes just one minute to register.

Dismiss Notice

Join WebmasterServe Forums 
Join the discussion! Have a better idea or an opinion? It takes just one minute to register Click Here to Join

Help Me/Question Where I Can Submit Robot.txt File?

Discussion in 'General Business Topics' started by Asifur Rahman Rakib, Feb 13, 2016.

  1. Asifur Rahman Rakib

    Yellow Belt

    Joined:
    Dec 31, 2015
    Messages:
    265
    Ratings:
    +109 / -0
    Hi guys,
    I have created robot.txt file for my blog posts .But Where do I submit this ? I don`t know.Please help me..I hope that any person help me.
     
  2. niranjan kumar

    Yellow Belt

    Joined:
    Feb 12, 2016
    Messages:
    471
    Ratings:
    +12 / -0
    As you want to know where you can put, first you need to know about Robots.txt. It is used for crawler and spiders, Because search engine understand better you website. Robots.txt not easy to make it's need some experiences for making robots.txt.
    You put robots.txt on your server home directory.
     
  3. RADHESHYAM GUPTA

    Yellow Belt

    Joined:
    Feb 13, 2016
    Messages:
    362
    Ratings:
    +20 / -0
    Robots.txt file is placed in your www or public_html directory. And you can Download your edited robots.txt code from the robots.txt Tester page by clicking Download in the Submit dialog. Upload your new robots.txt file to the root of your domain as a text file named robots.txt (the URL for your robots.txt file should be /robots.txt).
     
  4. John Philip

    White Belt

    Joined:
    Jul 25, 2018
    Messages:
    114
    Ratings:
    +0 / -0
  5. Yuva12

    White Belt

    Joined:
    Nov 23, 2018
    Messages:
    135
    Ratings:
    +1 / -0
    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.
     
  6. RH-Calvin

    Red Belt

    Joined:
    Jun 4, 2013
    Messages:
    671
    Ratings:
    +13 / -0
    Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.
     
  7. Monica Chortle

    White Belt

    Joined:
    Aug 4, 2018
    Messages:
    109
    Ratings:
    +3 / -0
    Robot.txt file would be placed at the roots of the website host that it applies.
     
  8. rajdeep7830

    White Belt

    Joined:
    Jan 30, 2019
    Messages:
    55
    Ratings:
    +0 / -0
    Robots.txt file is placed in your www or public_html directory.
     
  9. Onzmcn

    Onzmcn New Member

    Joined:
    Mar 25, 2019
    Messages:
    10
    Ratings:
    +0 / -0
    The robots.txt file should be located in the root directory of the site, but its presence on the site is not mandatory. It is designed to determine which pages of the site should exclude their processing and indexing by search engines. Or specify which search engine robots should or should not index the site. A standard robots.txt file might look like this:
    Code:
    Select All
    User-agent: * Disallow: / pages / Host: yoursite.com Sitemap: http: //yoursite/sitemap.xml
    The directive User-agent: * means that all robots are allowed to visit the site.
    The Disallow directive allows you to specify which pages to exclude from indexing. In the directive Host you specify the main address of your site. In addition, you can specify a sitemap file in this file.
     
  10. lewisclark019

    White Belt

    Joined:
    Dec 26, 2018
    Messages:
    76
    Ratings:
    +0 / -0
    The robots.txt file must be located at the root of the website host that it applies to. And If you have a WordPress Site then you need to install Yoast plugin and connected the Google search console.
     

Share This Page