Welcome to WebmasterServe!

FREE TO JOIN! Join us now to engage in informative and friendly discussions about Webmastering, SEO, SEM, Internet Marketing, Programming, Graphic Design, Online Jobs and more. What are you waiting for? Ready to join our friendly community? It takes just one minute to register.

Join Us
Dismiss Notice

Join WebmasterServe Forums 
Join the discussion! Have a better idea or an opinion? It takes just one minute to register Click Here to Join

Web Crawlers, Bots and Robots.txt

Discussions about managing web crawlers, bots using tools like robots.txt

Filter by Prefix:

Show All My Experience (2) Help Me/Question (16) Tutorial (1)
Best Posts
Sort By:
Title Start Date
Replies Views
Last Message ↓
  1. 100Bricks

    Help Me/Question What is robots.txt?

    100Bricks, May 26, 2015 at 1:15 AM ... 2
    Replies:
    25
    Views:
    895
    neelseowork
    Apr 7, 2018 at 12:41 AM
  2. 100Bricks

    Help Me/Question What is web crawler?

    100Bricks, May 26, 2015 at 1:13 AM ... 2
    Replies:
    33
    Views:
    886
    durgamatha
    Apr 6, 2018 at 1:31 AM
  3. Voora

    Help Me/Question Sitemap Generate, It Lists My Urls Twice

    Voora, Oct 11, 2017 at 2:21 AM
    Replies:
    0
    Views:
    44
    Voora
    Oct 11, 2017 at 2:21 AM
  4. Furqan Rashid

    My Experience Web Crawlers, Bots.

    Furqan Rashid, Jan 5, 2016 at 10:17 PM
    Replies:
    11
    Views:
    407
    RH-Calvin
    Aug 9, 2017 at 12:23 AM
  5. niranjan kumar

    Help Me/Question How Make Robots.txt

    niranjan kumar, Feb 14, 2016 at 4:08 AM
    Replies:
    7
    Views:
    184
    neelseofast
    Jul 27, 2017 at 4:45 AM
  6. Zirkon Kalti

    Tutorial How Does The Search Engine Work

    Zirkon Kalti, Feb 10, 2016 at 10:49 AM
    Replies:
    6
    Views:
    228
    Raj Gawdi
    Jul 27, 2017 at 2:23 AM
  7. Asifur Rahman Rakib

    Help Me/Question How Can I Create Robot.txt File Easily?

    Asifur Rahman Rakib, Feb 9, 2016 at 12:19 PM
    Replies:
    14
    Views:
    295
    neelseofast
    Jul 14, 2017 at 12:51 AM
  8. 100Bricks

    Help Me/Question What is a search engine bot?

    100Bricks, May 26, 2015 at 1:14 AM ... 2
    Replies:
    24
    Views:
    676
    mouni
    Jul 4, 2017 at 11:37 PM
  9. steve taylor

    Help Me/Question How To Stop Bad And Ugly Crawlers?

    steve taylor, Mar 4, 2016 at 4:45 AM
    Replies:
    6
    Views:
    252
    Manish Mishra
    Mar 17, 2016 at 1:08 AM
  10. Doominic anderson

    Help Me/Question If Web Crawlers Are Not Used By Search Engines

    Doominic anderson, Feb 6, 2016 at 7:17 AM
    Replies:
    4
    Views:
    144
    Swati Mishra
    Feb 28, 2016 at 6:51 AM
  11. Machostindia
    Locked

    Help Me/Question Showing google description error

    Machostindia, Dec 25, 2012 at 2:02 AM
    Replies:
    4
    Views:
    1,151
    Deepak Kaushik
    Dec 14, 2013 at 12:52 AM
  12. lapunluyang
    Locked

    My Experience Yahoo hasnt spidered my site after submitting it to them 1 month ago.

    lapunluyang, Apr 29, 2011 at 5:38 AM
    Replies:
    3
    Views:
    1,317
    Nittish Sharma
    Oct 15, 2013 at 4:57 AM
  13. imnikhilgavade
    Locked

    Help Me/Question 404 Crawl Errors Issue

    imnikhilgavade, Jun 4, 2013 at 2:32 AM
    Replies:
    2
    Views:
    433
    Evgeniy
    Jun 4, 2013 at 5:13 AM
  14. nasirkamal
    Locked

    Help Me/Question What Is The Benefits Of Google Sitemap?

    nasirkamal, Dec 18, 2012 at 6:41 AM
    Replies:
    7
    Views:
    846
    Machostindia
    Jan 9, 2013 at 5:23 AM
  15. bobsmith
    Locked

    Help Me/Question Site Map

    bobsmith, Jul 12, 2012 at 11:25 AM
    Replies:
    3
    Views:
    395
    macbuilders
    Sep 8, 2012 at 8:44 AM
  16. mileycullens
    Locked

    Help Me/Question Can Anyone Help About The Correct Format Of Robots.txt?

    mileycullens, Apr 23, 2010 at 7:38 AM
    Replies:
    5
    Views:
    998
    softwaredevelopersuk
    Apr 23, 2012 at 5:34 AM
  17. Alex997
    Locked

    Help Me/Question Multiple Sitemaps, Single Domain - Overlap Penalties?

    Alex997, Jan 25, 2012 at 10:14 AM
    Replies:
    1
    Views:
    861
    Webmasterserve
    Jan 26, 2012 at 3:15 AM
  18. clerisy
    Locked

    Help Me/Question What Is Site Map And Robots.txt?

    clerisy, Jan 21, 2012 at 12:33 AM
    Replies:
    1
    Views:
    436
    commodityt
    Jan 21, 2012 at 12:56 AM
  19. gkd_uk
    Locked

    Help Me/Question How Do You Control Spamming On Your Blog?

    gkd_uk, Jun 16, 2011 at 3:10 PM
    Replies:
    4
    Views:
    767
    TailoredVPS
    Jun 21, 2011 at 7:17 PM
Showing threads 1 to 19 of 19

Thread Display Options

Loading...
(Please sign-up or login to post here)