Welcome to WebmasterServe!

FREE TO JOIN! Join us now to engage in informative and friendly discussions about Webmastering, SEO, SEM, Internet Marketing, Programming, Graphic Design, Online Jobs and more. What are you waiting for? Ready to join our friendly community? It takes just one minute to register.

Join Us
Dismiss Notice

Join WebmasterServe Forums 
Join the discussion! Have a better idea or an opinion? It takes just one minute to register Click Here to Join

Web Crawlers, Bots and Robots.txt

Discussions about managing web crawlers, bots using tools like robots.txt

Filter by Prefix:

Show All My Experience (2) Help Me/Question (15) Tutorial (1)
Best Posts
Sort By:
Title Start Date
Replies Views
Last Message ↓
  1. 100Bricks

    Help Me/Question What is a search engine bot?

    100Bricks, May 26, 2015 at 1:14 AM ... 2
    Replies:
    20
    Views:
    400
    geltimer
    Jun 17, 2017 at 4:21 AM
  2. Furqan Rashid

    My Experience Web Crawlers, Bots.

    Furqan Rashid, Jan 5, 2016 at 10:17 PM
    Replies:
    9
    Views:
    138
    Patricia Mangum
    Jun 1, 2017 at 6:19 AM
  3. niranjan kumar

    Help Me/Question How Make Robots.txt

    niranjan kumar, Feb 14, 2016 at 4:08 AM
    Replies:
    2
    Views:
    62
    RH-Calvin
    May 11, 2017 at 1:10 AM
  4. 100Bricks

    Help Me/Question What is web crawler?

    100Bricks, May 26, 2015 at 1:13 AM ... 2
    Replies:
    24
    Views:
    555
    ademar
    Apr 29, 2017 at 1:48 AM
  5. Asifur Rahman Rakib

    Help Me/Question How Can I Create Robot.txt File Easily?

    Asifur Rahman Rakib, Feb 9, 2016 at 12:19 PM
    Replies:
    10
    Views:
    122
    sin123
    Mar 2, 2017 at 11:46 PM
  6. 100Bricks

    Help Me/Question What is robots.txt?

    100Bricks, May 26, 2015 at 1:15 AM
    Replies:
    14
    Views:
    568
    genuinework789
    Feb 23, 2017 at 11:51 PM
  7. steve taylor

    Help Me/Question How To Stop Bad And Ugly Crawlers?

    steve taylor, Mar 4, 2016 at 4:45 AM
    Replies:
    6
    Views:
    98
    Manish Mishra
    Mar 17, 2016 at 1:08 AM
  8. Doominic anderson

    Help Me/Question If Web Crawlers Are Not Used By Search Engines

    Doominic anderson, Feb 6, 2016 at 7:17 AM
    Replies:
    4
    Views:
    58
    Swati Mishra
    Feb 28, 2016 at 6:51 AM
  9. Zirkon Kalti

    Tutorial How Does The Search Engine Work

    Zirkon Kalti, Feb 10, 2016 at 10:49 AM
    Replies:
    0
    Views:
    62
    Zirkon Kalti
    Feb 10, 2016 at 10:49 AM
  10. Machostindia
    Locked

    Help Me/Question Showing google description error

    Machostindia, Dec 25, 2012 at 2:02 AM
    Replies:
    4
    Views:
    1,054
    Deepak Kaushik
    Dec 14, 2013 at 12:52 AM
  11. lapunluyang
    Locked

    My Experience Yahoo hasnt spidered my site after submitting it to them 1 month ago.

    lapunluyang, Apr 29, 2011 at 5:38 AM
    Replies:
    3
    Views:
    1,265
    Nittish Sharma
    Oct 15, 2013 at 4:57 AM
  12. imnikhilgavade
    Locked

    Help Me/Question 404 Crawl Errors Issue

    imnikhilgavade, Jun 4, 2013 at 2:32 AM
    Replies:
    2
    Views:
    404
    Evgeniy
    Jun 4, 2013 at 5:13 AM
  13. nasirkamal
    Locked

    Help Me/Question What Is The Benefits Of Google Sitemap?

    nasirkamal, Dec 18, 2012 at 6:41 AM
    Replies:
    7
    Views:
    788
    Machostindia
    Jan 9, 2013 at 5:23 AM
  14. bobsmith
    Locked

    Help Me/Question Site Map

    bobsmith, Jul 12, 2012 at 11:25 AM
    Replies:
    3
    Views:
    343
    macbuilders
    Sep 8, 2012 at 8:44 AM
  15. mileycullens
    Locked

    Help Me/Question Can Anyone Help About The Correct Format Of Robots.txt?

    mileycullens, Apr 23, 2010 at 7:38 AM
    Replies:
    5
    Views:
    962
    softwaredevelopersuk
    Apr 23, 2012 at 5:34 AM
  16. Alex997
    Locked

    Help Me/Question Multiple Sitemaps, Single Domain - Overlap Penalties?

    Alex997, Jan 25, 2012 at 10:14 AM
    Replies:
    1
    Views:
    801
    Webmasterserve
    Jan 26, 2012 at 3:15 AM
  17. clerisy
    Locked

    Help Me/Question What Is Site Map And Robots.txt?

    clerisy, Jan 21, 2012 at 12:33 AM
    Replies:
    1
    Views:
    404
    commodityt
    Jan 21, 2012 at 12:56 AM
  18. gkd_uk
    Locked

    Help Me/Question How Do You Control Spamming On Your Blog?

    gkd_uk, Jun 16, 2011 at 3:10 PM
    Replies:
    4
    Views:
    709
    TailoredVPS
    Jun 21, 2011 at 7:17 PM
Showing threads 1 to 18 of 18

Thread Display Options

Loading...
(Please sign-up or login to post here)