Welcome to WebmasterServe!

FREE TO JOIN! Join us now to engage in informative and friendly discussions about Webmastering, SEO, SEM, Internet Marketing, Programming, Graphic Design, Online Jobs and more. What are you waiting for? Ready to join our friendly community? It takes just one minute to register.

Join Us
Dismiss Notice

Join WebmasterServe Forums 
Join the discussion! Have a better idea or an opinion? It takes just one minute to register Click Here to Join

Web Crawlers, Bots and Robots.txt

Discussions about managing web crawlers, bots using tools like robots.txt

Filter by Prefix:

Show All My Experience (2) Help Me/Question (15) Tutorial (1)
Best Posts
Sort By:
Title Start Date
Replies Views
Last Message ↓
  1. Furqan Rashid

    My Experience Web Crawlers, Bots.

    Furqan Rashid, Jan 5, 2016 at 10:17 PM
    Replies:
    11
    Views:
    196
    RH-Calvin
    Aug 9, 2017 at 12:23 AM
  2. 100Bricks

    Help Me/Question What is robots.txt?

    100Bricks, May 26, 2015 at 1:15 AM ... 2
    Replies:
    20
    Views:
    641
    Give Me Challenge
    Aug 8, 2017 at 12:52 AM
  3. niranjan kumar

    Help Me/Question How Make Robots.txt

    niranjan kumar, Feb 14, 2016 at 4:08 AM
    Replies:
    7
    Views:
    103
    neelseofast
    Jul 27, 2017 at 4:45 AM
  4. Zirkon Kalti

    Tutorial How Does The Search Engine Work

    Zirkon Kalti, Feb 10, 2016 at 10:49 AM
    Replies:
    6
    Views:
    136
    Raj Gawdi
    Jul 27, 2017 at 2:23 AM
  5. Asifur Rahman Rakib

    Help Me/Question How Can I Create Robot.txt File Easily?

    Asifur Rahman Rakib, Feb 9, 2016 at 12:19 PM
    Replies:
    14
    Views:
    190
    neelseofast
    Jul 14, 2017 at 12:51 AM
  6. 100Bricks

    Help Me/Question What is a search engine bot?

    100Bricks, May 26, 2015 at 1:14 AM ... 2
    Replies:
    24
    Views:
    469
    mouni
    Jul 4, 2017 at 11:37 PM
  7. 100Bricks

    Help Me/Question What is web crawler?

    100Bricks, May 26, 2015 at 1:13 AM ... 2
    Replies:
    24
    Views:
    592
    ademar
    Apr 29, 2017 at 1:48 AM
  8. steve taylor

    Help Me/Question How To Stop Bad And Ugly Crawlers?

    steve taylor, Mar 4, 2016 at 4:45 AM
    Replies:
    6
    Views:
    114
    Manish Mishra
    Mar 17, 2016 at 1:08 AM
  9. Doominic anderson

    Help Me/Question If Web Crawlers Are Not Used By Search Engines

    Doominic anderson, Feb 6, 2016 at 7:17 AM
    Replies:
    4
    Views:
    71
    Swati Mishra
    Feb 28, 2016 at 6:51 AM
  10. Machostindia
    Locked

    Help Me/Question Showing google description error

    Machostindia, Dec 25, 2012 at 2:02 AM
    Replies:
    4
    Views:
    1,064
    Deepak Kaushik
    Dec 14, 2013 at 12:52 AM
  11. lapunluyang
    Locked

    My Experience Yahoo hasnt spidered my site after submitting it to them 1 month ago.

    lapunluyang, Apr 29, 2011 at 5:38 AM
    Replies:
    3
    Views:
    1,268
    Nittish Sharma
    Oct 15, 2013 at 4:57 AM
  12. imnikhilgavade
    Locked

    Help Me/Question 404 Crawl Errors Issue

    imnikhilgavade, Jun 4, 2013 at 2:32 AM
    Replies:
    2
    Views:
    405
    Evgeniy
    Jun 4, 2013 at 5:13 AM
  13. nasirkamal
    Locked

    Help Me/Question What Is The Benefits Of Google Sitemap?

    nasirkamal, Dec 18, 2012 at 6:41 AM
    Replies:
    7
    Views:
    798
    Machostindia
    Jan 9, 2013 at 5:23 AM
  14. bobsmith
    Locked

    Help Me/Question Site Map

    bobsmith, Jul 12, 2012 at 11:25 AM
    Replies:
    3
    Views:
    346
    macbuilders
    Sep 8, 2012 at 8:44 AM
  15. mileycullens
    Locked

    Help Me/Question Can Anyone Help About The Correct Format Of Robots.txt?

    mileycullens, Apr 23, 2010 at 7:38 AM
    Replies:
    5
    Views:
    965
    softwaredevelopersuk
    Apr 23, 2012 at 5:34 AM
  16. Alex997
    Locked

    Help Me/Question Multiple Sitemaps, Single Domain - Overlap Penalties?

    Alex997, Jan 25, 2012 at 10:14 AM
    Replies:
    1
    Views:
    804
    Webmasterserve
    Jan 26, 2012 at 3:15 AM
  17. clerisy
    Locked

    Help Me/Question What Is Site Map And Robots.txt?

    clerisy, Jan 21, 2012 at 12:33 AM
    Replies:
    1
    Views:
    408
    commodityt
    Jan 21, 2012 at 12:56 AM
  18. gkd_uk
    Locked

    Help Me/Question How Do You Control Spamming On Your Blog?

    gkd_uk, Jun 16, 2011 at 3:10 PM
    Replies:
    4
    Views:
    717
    TailoredVPS
    Jun 21, 2011 at 7:17 PM
Showing threads 1 to 18 of 18

Thread Display Options

Loading...
(Please sign-up or login to post here)