Help Me/Question What is web crawler?


White Belt
A crawler is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index.


Yellow Belt
A crawler is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index.


White Belt
A crawler is a relatively simple automated program that methodically scans through Internet pages to create an index of the data.


Yellow Belt
Web crawlers are nothing but programs which helps to collect relevant information for the search engines by crawling the web. There are many web crawlers some of them are:
1. Google crawler
2. Yahoo crawler
3. MSN crawler
4. Web RACE

Zirkon Kalti

Content Writer
Web crawler, also known as spider and bots, is a program designed to scan the web for anything they could find. Search engine regularly dispatch web crawler to scan the web for new content to be added into the index. When someone search for a keyword on the search engine, the web crawler will search its index for websites that have relevant content that match with the keyword and return it in the search result.


Blue Belt
A web crawler, second name “web spider” or “web robot” is a software program or robotic script which browses our websites automatically in a systematic way. Simply this procedure is called web spidering or crawling. Its purpose is usually just to index the websites and cached its contents. There are a lot of legitimate sites, in specific search engines, use web spider as a means of providing current published data.


Active Member
the web pages of any sites walking into the search engine is said to be crawling of the web pages into the search engine

Manish Mishra

Content Writer
To making an index of new or existing page/pages of a website requires a process that is called crawling.This process is done by web crawler.
Basically, it is software that works automatically to target web content used in web indexing system. This process is done by the help of search engine bots.

Please read: If requires further: Help Me/Question - What is a search engine bot?


White Belt
There are various major search engine and every search engine has a crawler, And crawler is also known as Spider or bot.
A web crawler is visit on website for read your content for collecting information or keywords, it's also known as web crawler.
This process is call crawling or web crawling.


Content Writer
web crawler is a type for software programs runs at the back end of each search engine. The purpose of search engine is to crawl and update the database over a period of time. It contains a large logic's behind them.

Swati Mishra

Content Writer
we can say web crawler as web robot or web spider that is the program which browses the WWW ( world wide web) in automated manner.

steve taylor

White Belt
A crawler is a program visit new websites and reads all the pages and other information in order to create entries for a search engine index. The main search engine on the web all contain a such a program. It is also known as spider or bot . crawlers are generally developed to visit sites that is submitted by their owners as new or updated.

niranjan kumar

White Belt
Web crawler is a automated software that work for search engines . Mainly web crawler works is go to a lot of websites in same time they read about websites.
They also read about your website sitemap and crawler is most import for getting higher rank in search engine rank page. if you want to know more about crawler follower some blogs.


Money Making Ideas Online UAE, UK, USA
Web crawler is used to automatically update your new data on search engine fast.
A web crawler is a program or automated script which browses the World Wide Web in a methodical, automated manner. This process is called Web crawling or spidering. Many legitimate sites, in particular search engines, use spidering as a means of providing up-to-date data.