%0 Journal Article
%T Web Crawler: A Review
%A Dhiraj Khurana
%A Satish Kumar
%J International Journal of Computer Science and Management Studies
%D 2012
%I Imperial Foundation
%X In a large distributed system like the Web, users find resources by following hypertext links from one document to another. When the system is small and its resources share the same fundamental purpose, users can find resources of interest with relative ease.However, with the Web now encompassing millions of sites with many different purposes, navigation is difficult. WebCrawler, the WebĄ¯s first comprehensive full-text search engine, is a tool that assists users in their Web navigation by automating the task of linktraversal, creating a searchable index of the web, and fulfilling searchersĄ¯ queries from the index. Conceptually, WebCrawler is a node in the Web graph that contains links to many sites on the net, shortening the path between users and their destinations.
%K Crawler
%K Optimization
%K Duplicate
%U http://www.ijcsms.com/journals/Volume%2012,%20Issue%2001,%20January%202012_WEB_CRAWLER1.pdf