Help Me/Question How To Stop Bad And Ugly Crawlers?

steve taylor

White Belt
As we know crawlers is a computer program and it is written for gathering useful information from new websites which information helps search engine to build index. But there are also some bad web crawlers are exists that come to website and consume the website resources like website traffic and gather information illegally.
 

Zirkon Kalti

Content Writer
You can stop the search engine crawlers from crawling your site so frequently by making some adjustment in the robots.txt. You can use the term crawl-delay to delay to increase the time the search engine takes to crawl the entire site. For example, the search engine will take about 6 days to crawl the entire site if you set a crawl delay of 500 seconds in the robots.txt. You can also disallow folders that you don't want the search engine crawlers to crawl. If the crawling rate did not exceed 1000 per day, there is no need to make any adjustment.
 

niranjan kumar

White Belt
There are many crawler some are good and some are used for bad purpose, when you are making robots.txt for your website. You can set google boat or crawler, how many crawler can index your webpage. so make a robots.txt according to you that have only good crawlers are allowed and bad crawlers are disallowed.
 

RADHESHYAM GUPTA

White Belt
There are so many search engine and every search engine has crawler some of these are very important like google crawler and some of are bad.
If you want to stop bad and ugly crawler just create a Robot.txt file in your website and create your program which content or webpage is allow for the crawler.
 

Doominic anderson

White Belt
Crawler of the search engine responsible for indexing and search the keyword inserted by the user of that search engine.
If you want to stop the bad crawlers to indexing your website in it's database then you have to write some code in the head section of your html document that for which search you want to give the do follow option,if you want to restrict such type of search engine you have to just write a code of no follow for these types of search engines.
 

sunshine

Blue Belt
Web Crawlers always love the good, but kill the bad and the ugly bots. Google (by far) and Bing (to a minor extend) have the most efficient web crawlers and do not respect the robots.txt for the crawl-delay settings. You have to adjust their crawl rate manually.
However, use META tags to block robots. You can try adding meta tags to the head of each web page. For example:
<html>
<head>
<title>...</title>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>
 

Manish Mishra

Content Writer
Web Crawlers always love the good, but kill the bad and the ugly bots. Google (by far) and Bing (to a minor extend) have the most efficient web crawlers and do not respect the robots.txt for the crawl-delay settings. You have to adjust their crawl rate manually.
However, use META tags to block robots. You can try adding meta tags to the head of each web page. For example:
<html>
<head>
<title>...</title>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>
@sunshine: Thanks mate you have helped with this great post.
If you do not want to get followed by bad bots just let them not to but i don't believe if they will hurt your ranking by any means.
 
Top